2024-03-25T16:11:40.386 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-25T16:11:40.386 DEBUG:teuthology.run:Teuthology command: teuthology --name amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi --archive /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621258 --description rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} --owner scheduled_amathuri@teuthology --verbose -- /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621258/orig.config.yaml 2024-03-25T16:11:40.429 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-25T16:11:40.521 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621258 branch: main description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} email: null first_in_suite: false job_id: '7621258' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: client: debug monc: 20 debug ms: 1 debug objecter: 20 debug rados: 20 global: mon client directed command retry: 5 mon election default strategy: 3 ms bind msgr2: false ms inject socket failures: 5000 ms type: async osd heartbeat grace: 80 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osd crush smoke test: false mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 mon warn on pool no app: false paxos service trim min: 10 osd: bdev async discard: true bdev enable discard: true bluestore allocator: stupid bluestore block size: 96636764160 bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 1 debug bluestore: 1 debug monc: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd blocked scrub grace period: 3600 osd class default list: '*' osd class load list: '*' osd client watch timeout: 120 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd fast shutdown: false osd max markdown count: 1000 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - but it is still running - objects unfound and apparently lost - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN - PG_DEGRADED - PG_AVAILABILITY - POOL_APP_NOT_ENABLED - \(OSD_SLOW_PING_TIME - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(SLOW_OPS\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(OBJECT_MISPLACED\) - slow request - overall HEALTH_ - \(PG_ - \(MON_DOWN\) - \(SLOW_OPS\) - slow request mon_bind_msgr2: false sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 valgrind: mds: - --tool=memcheck mon: - --tool=memcheck - --leak-check=full - --show-reachable=yes osd: - --tool=memcheck ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: debuginfo: true flavor: default sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: main sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 owner: scheduled_amathuri@teuthology priority: 150 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 6785 sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 sleep_before_teardown: 0 subset: 99/1000 suite: rados:verify suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 targets: smithi097.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAlUFkA3EhV5kb2ygNpF5qBDAejAgsqx9lIlhqLrnXLtG4ypJAUAbWNTJwBXqf2oFG5BOn1WPktEm7I8kPXy5/4= smithi098.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBG5PMryiLKOLykE/21YnL5gEtLKYPstczuT3fgQ4kfaLo8aFaCr4KeWyCXgjdzw37FS+H3mKEFtX3vKa3yCpmuU= tasks: - install: null - ceph: null - thrashosds: chance_pgnum_grow: 1 chance_pgnum_shrink: 1 chance_pgpnum_fix: 1 timeout: 1200 - workunit: clients: client.0: - rados/test.sh env: ALLOW_TIMEOUTS: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-25_13:28:33 tube: smithi user: amathuri verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-25T16:11:40.521 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa; will attempt to use it 2024-03-25T16:11:40.522 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks 2024-03-25T16:11:40.522 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-25T16:11:40.523 INFO:teuthology.task.internal:Checking packages... 2024-03-25T16:11:40.548 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '5dcb190da270440ef141d088eb27a9fdb8afc5f2' 2024-03-25T16:11:40.548 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-25T16:11:40.548 INFO:teuthology.packaging:ref: None 2024-03-25T16:11:40.549 INFO:teuthology.packaging:tag: None 2024-03-25T16:11:40.549 INFO:teuthology.packaging:branch: main 2024-03-25T16:11:40.549 INFO:teuthology.packaging:sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T16:11:40.549 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-03-25T16:11:40.779 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2381.g44a9b520 2024-03-25T16:11:40.781 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-25T16:11:40.788 INFO:teuthology.task.internal:no buildpackages task found 2024-03-25T16:11:40.788 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-25T16:11:40.887 INFO:teuthology.task.internal:Saving configuration 2024-03-25T16:11:40.904 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-25T16:11:40.928 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T16:11:40.950 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi097.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621258', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 16:06:47.297466', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAlUFkA3EhV5kb2ygNpF5qBDAejAgsqx9lIlhqLrnXLtG4ypJAUAbWNTJwBXqf2oFG5BOn1WPktEm7I8kPXy5/4='} 2024-03-25T16:11:40.972 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi098.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621258', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 16:06:47.295486', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBG5PMryiLKOLykE/21YnL5gEtLKYPstczuT3fgQ4kfaLo8aFaCr4KeWyCXgjdzw37FS+H3mKEFtX3vKa3yCpmuU='} 2024-03-25T16:11:40.972 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-25T16:11:40.978 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2024-03-25T16:11:40.978 INFO:teuthology.task.internal:roles: ubuntu@smithi098.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-03-25T16:11:40.978 INFO:teuthology.run_tasks:Running task console_log... 2024-03-25T16:11:41.071 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7ff1e81fa0d0>, signals=[15]) 2024-03-25T16:11:41.071 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-25T16:11:41.077 INFO:teuthology.task.internal:Opening connections... 2024-03-25T16:11:41.077 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi097.front.sepia.ceph.com 2024-03-25T16:11:41.079 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi097.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:11:41.171 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi098.front.sepia.ceph.com 2024-03-25T16:11:41.173 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:11:41.260 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-25T16:11:41.266 DEBUG:teuthology.orchestra.run.smithi097:> uname -m 2024-03-25T16:11:41.283 INFO:teuthology.orchestra.run.smithi097.stdout:x86_64 2024-03-25T16:11:41.284 DEBUG:teuthology.orchestra.run.smithi097:> cat /etc/os-release 2024-03-25T16:11:41.340 INFO:teuthology.orchestra.run.smithi097.stdout:NAME="CentOS Stream" 2024-03-25T16:11:41.340 INFO:teuthology.orchestra.run.smithi097.stdout:VERSION="9" 2024-03-25T16:11:41.341 INFO:teuthology.orchestra.run.smithi097.stdout:ID="centos" 2024-03-25T16:11:41.341 INFO:teuthology.orchestra.run.smithi097.stdout:ID_LIKE="rhel fedora" 2024-03-25T16:11:41.341 INFO:teuthology.orchestra.run.smithi097.stdout:VERSION_ID="9" 2024-03-25T16:11:41.341 INFO:teuthology.orchestra.run.smithi097.stdout:PLATFORM_ID="platform:el9" 2024-03-25T16:11:41.341 INFO:teuthology.orchestra.run.smithi097.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-25T16:11:41.341 INFO:teuthology.orchestra.run.smithi097.stdout:ANSI_COLOR="0;31" 2024-03-25T16:11:41.341 INFO:teuthology.orchestra.run.smithi097.stdout:LOGO="fedora-logo-icon" 2024-03-25T16:11:41.341 INFO:teuthology.orchestra.run.smithi097.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-25T16:11:41.341 INFO:teuthology.orchestra.run.smithi097.stdout:HOME_URL="https://centos.org/" 2024-03-25T16:11:41.341 INFO:teuthology.orchestra.run.smithi097.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-25T16:11:41.341 INFO:teuthology.orchestra.run.smithi097.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-25T16:11:41.341 INFO:teuthology.orchestra.run.smithi097.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-25T16:11:41.342 INFO:teuthology.lock.ops:Updating smithi097.front.sepia.ceph.com on lock server 2024-03-25T16:11:41.365 DEBUG:teuthology.orchestra.run.smithi098:> uname -m 2024-03-25T16:11:41.382 INFO:teuthology.orchestra.run.smithi098.stdout:x86_64 2024-03-25T16:11:41.383 DEBUG:teuthology.orchestra.run.smithi098:> cat /etc/os-release 2024-03-25T16:11:41.440 INFO:teuthology.orchestra.run.smithi098.stdout:NAME="CentOS Stream" 2024-03-25T16:11:41.440 INFO:teuthology.orchestra.run.smithi098.stdout:VERSION="9" 2024-03-25T16:11:41.440 INFO:teuthology.orchestra.run.smithi098.stdout:ID="centos" 2024-03-25T16:11:41.440 INFO:teuthology.orchestra.run.smithi098.stdout:ID_LIKE="rhel fedora" 2024-03-25T16:11:41.440 INFO:teuthology.orchestra.run.smithi098.stdout:VERSION_ID="9" 2024-03-25T16:11:41.440 INFO:teuthology.orchestra.run.smithi098.stdout:PLATFORM_ID="platform:el9" 2024-03-25T16:11:41.440 INFO:teuthology.orchestra.run.smithi098.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-25T16:11:41.440 INFO:teuthology.orchestra.run.smithi098.stdout:ANSI_COLOR="0;31" 2024-03-25T16:11:41.440 INFO:teuthology.orchestra.run.smithi098.stdout:LOGO="fedora-logo-icon" 2024-03-25T16:11:41.440 INFO:teuthology.orchestra.run.smithi098.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-25T16:11:41.440 INFO:teuthology.orchestra.run.smithi098.stdout:HOME_URL="https://centos.org/" 2024-03-25T16:11:41.440 INFO:teuthology.orchestra.run.smithi098.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-25T16:11:41.441 INFO:teuthology.orchestra.run.smithi098.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-25T16:11:41.441 INFO:teuthology.orchestra.run.smithi098.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-25T16:11:41.441 INFO:teuthology.lock.ops:Updating smithi098.front.sepia.ceph.com on lock server 2024-03-25T16:11:41.465 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-25T16:11:41.474 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-25T16:11:41.495 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-25T16:11:41.496 DEBUG:teuthology.orchestra.run.smithi097:> test '!' -e /home/ubuntu/cephtest 2024-03-25T16:11:41.498 DEBUG:teuthology.orchestra.run.smithi098:> test '!' -e /home/ubuntu/cephtest 2024-03-25T16:11:41.515 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-25T16:11:41.520 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-25T16:11:41.520 DEBUG:teuthology.orchestra.run.smithi097:> test -z $(ls -A /var/lib/ceph) 2024-03-25T16:11:41.555 DEBUG:teuthology.orchestra.run.smithi098:> test -z $(ls -A /var/lib/ceph) 2024-03-25T16:11:41.578 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-25T16:11:41.627 INFO:teuthology.run_tasks:Running task kernel... 2024-03-25T16:11:41.642 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-25T16:11:41.642 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-25T16:11:41.642 DEBUG:teuthology.orchestra.run.smithi097:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-25T16:11:41.643 DEBUG:teuthology.orchestra.run.smithi098:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-25T16:11:41.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T16:11:41.659 DEBUG:teuthology.orchestra.run.smithi097:> uname -r 2024-03-25T16:11:41.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T16:11:41.660 DEBUG:teuthology.orchestra.run.smithi098:> uname -r 2024-03-25T16:11:41.716 INFO:teuthology.orchestra.run.smithi097.stdout:5.14.0-361.el9.x86_64 2024-03-25T16:11:41.716 INFO:teuthology.task.kernel:Running kernel on smithi097: 5.14.0-361.el9.x86_64 2024-03-25T16:11:41.717 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum install -y kernel 2024-03-25T16:11:41.718 INFO:teuthology.orchestra.run.smithi098.stdout:5.14.0-361.el9.x86_64 2024-03-25T16:11:41.718 INFO:teuthology.task.kernel:Running kernel on smithi098: 5.14.0-361.el9.x86_64 2024-03-25T16:11:41.718 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum install -y kernel 2024-03-25T16:11:44.281 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS-9 - AppStream 7.2 MB/s | 6.7 MB 00:00 2024-03-25T16:11:44.290 INFO:teuthology.orchestra.run.smithi097.stdout:CentOS-9 - AppStream 7.5 MB/s | 6.7 MB 00:00 2024-03-25T16:11:45.914 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS-9 - BaseOS 3.8 MB/s | 2.0 MB 00:00 2024-03-25T16:11:45.940 INFO:teuthology.orchestra.run.smithi097.stdout:CentOS-9 - BaseOS 3.9 MB/s | 2.0 MB 00:00 2024-03-25T16:11:46.730 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - BaseOS 17 MB/s | 8.0 MB 00:00 2024-03-25T16:11:47.822 INFO:teuthology.orchestra.run.smithi097.stdout:CentOS Stream 9 - BaseOS 5.6 MB/s | 8.0 MB 00:01 2024-03-25T16:11:48.591 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - AppStream 23 MB/s | 19 MB 00:00 2024-03-25T16:11:53.574 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - CRB 4.1 MB/s | 6.1 MB 00:01 2024-03-25T16:11:55.024 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - Extras packages 52 kB/s | 16 kB 00:00 2024-03-25T16:11:56.928 INFO:teuthology.orchestra.run.smithi098.stdout:Extra Packages for Enterprise Linux 12 MB/s | 21 MB 00:01 2024-03-25T16:12:01.817 INFO:teuthology.orchestra.run.smithi098.stdout:lab-extras 14 kB/s | 1.7 kB 00:00 2024-03-25T16:12:03.941 INFO:teuthology.orchestra.run.smithi098.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-25T16:12:03.941 INFO:teuthology.orchestra.run.smithi098.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-25T16:12:04.030 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-25T16:12:04.036 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T16:12:04.036 INFO:teuthology.orchestra.run.smithi098.stdout: Package Architecture Version Repository Size 2024-03-25T16:12:04.036 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T16:12:04.036 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-03-25T16:12:04.036 INFO:teuthology.orchestra.run.smithi098.stdout: kernel x86_64 5.14.0-430.el9 baseos 44 k 2024-03-25T16:12:04.036 INFO:teuthology.orchestra.run.smithi098.stdout:Installing dependencies: 2024-03-25T16:12:04.036 INFO:teuthology.orchestra.run.smithi098.stdout: kernel-core x86_64 5.14.0-430.el9 baseos 15 M 2024-03-25T16:12:04.036 INFO:teuthology.orchestra.run.smithi098.stdout: kernel-modules x86_64 5.14.0-430.el9 baseos 33 M 2024-03-25T16:12:04.036 INFO:teuthology.orchestra.run.smithi098.stdout: kernel-modules-core x86_64 5.14.0-430.el9 baseos 27 M 2024-03-25T16:12:04.036 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T16:12:04.036 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-03-25T16:12:04.037 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T16:12:04.037 INFO:teuthology.orchestra.run.smithi098.stdout:Install 4 Packages 2024-03-25T16:12:04.037 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T16:12:04.037 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 76 M 2024-03-25T16:12:04.037 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 123 M 2024-03-25T16:12:04.037 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-03-25T16:12:04.953 INFO:teuthology.orchestra.run.smithi098.stdout:(1/4): kernel-5.14.0-430.el9.x86_64.rpm 53 kB/s | 44 kB 00:00 2024-03-25T16:12:13.171 INFO:teuthology.orchestra.run.smithi097.stdout:CentOS Stream 9 - AppStream 795 kB/s | 19 MB 00:24 2024-03-25T16:12:24.829 INFO:teuthology.orchestra.run.smithi097.stdout:CentOS Stream 9 - CRB 763 kB/s | 6.1 MB 00:08 2024-03-25T16:12:25.626 INFO:teuthology.orchestra.run.smithi098.stdout:(2/4): kernel-core-5.14.0-430.el9.x86_64.rpm 709 kB/s | 15 MB 00:21 2024-03-25T16:12:26.538 INFO:teuthology.orchestra.run.smithi097.stdout:CentOS Stream 9 - Extras packages 26 kB/s | 16 kB 00:00 2024-03-25T16:12:27.439 INFO:teuthology.orchestra.run.smithi097.stdout:Extra Packages for Enterprise Linux 26 MB/s | 21 MB 00:00 2024-03-25T16:12:32.406 INFO:teuthology.orchestra.run.smithi097.stdout:lab-extras 14 kB/s | 1.7 kB 00:00 2024-03-25T16:12:34.512 INFO:teuthology.orchestra.run.smithi097.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-25T16:12:34.513 INFO:teuthology.orchestra.run.smithi097.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-25T16:12:34.648 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-03-25T16:12:34.658 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-03-25T16:12:34.659 INFO:teuthology.orchestra.run.smithi097.stdout: Package Architecture Version Repository Size 2024-03-25T16:12:34.659 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-03-25T16:12:34.659 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2024-03-25T16:12:34.659 INFO:teuthology.orchestra.run.smithi097.stdout: kernel x86_64 5.14.0-430.el9 baseos 44 k 2024-03-25T16:12:34.659 INFO:teuthology.orchestra.run.smithi097.stdout:Installing dependencies: 2024-03-25T16:12:34.659 INFO:teuthology.orchestra.run.smithi097.stdout: kernel-core x86_64 5.14.0-430.el9 baseos 15 M 2024-03-25T16:12:34.659 INFO:teuthology.orchestra.run.smithi097.stdout: kernel-modules x86_64 5.14.0-430.el9 baseos 33 M 2024-03-25T16:12:34.659 INFO:teuthology.orchestra.run.smithi097.stdout: kernel-modules-core x86_64 5.14.0-430.el9 baseos 27 M 2024-03-25T16:12:34.659 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T16:12:34.659 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2024-03-25T16:12:34.659 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-03-25T16:12:34.659 INFO:teuthology.orchestra.run.smithi097.stdout:Install 4 Packages 2024-03-25T16:12:34.660 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T16:12:34.660 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 76 M 2024-03-25T16:12:34.660 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 123 M 2024-03-25T16:12:34.660 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2024-03-25T16:12:35.024 INFO:teuthology.orchestra.run.smithi097.stdout:(1/4): kernel-5.14.0-430.el9.x86_64.rpm 192 kB/s | 44 kB 00:00 2024-03-25T16:12:35.491 INFO:teuthology.orchestra.run.smithi097.stdout:(2/4): kernel-core-5.14.0-430.el9.x86_64.rpm 21 MB/s | 15 MB 00:00 2024-03-25T16:12:36.666 INFO:teuthology.orchestra.run.smithi097.stdout:(3/4): kernel-modules-5.14.0-430.el9.x86_64.rpm 18 MB/s | 33 MB 00:01 2024-03-25T16:12:38.707 INFO:teuthology.orchestra.run.smithi097.stdout:(4/4): kernel-modules-core-5.14.0-430.el9.x86_6 7.5 MB/s | 27 MB 00:03 2024-03-25T16:12:38.710 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:12:38.710 INFO:teuthology.orchestra.run.smithi097.stdout:Total 19 MB/s | 76 MB 00:04 2024-03-25T16:12:39.373 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2024-03-25T16:12:39.730 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2024-03-25T16:12:39.730 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2024-03-25T16:12:40.775 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2024-03-25T16:12:40.776 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2024-03-25T16:12:43.244 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2024-03-25T16:12:43.901 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : kernel-modules-core-5.14.0-430.el9.x86_64 1/4 2024-03-25T16:12:43.999 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:12:44.341 INFO:teuthology.orchestra.run.smithi098.stdout:(3/4): kernel-modules-core-5.14.0-430.el9.x86_6 714 kB/s | 27 MB 00:39 2024-03-25T16:12:46.771 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:12:46.910 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:12:49.740 INFO:teuthology.orchestra.run.smithi098.stdout:(4/4): kernel-modules-5.14.0-430.el9.x86_64.rpm 751 kB/s | 33 MB 00:45 2024-03-25T16:12:49.741 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:12:49.741 INFO:teuthology.orchestra.run.smithi098.stdout:Total 1.7 MB/s | 76 MB 00:45 2024-03-25T16:12:50.406 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-03-25T16:12:50.760 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-03-25T16:12:50.761 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-03-25T16:12:51.610 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:12:51.652 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : kernel-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:12:51.758 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-03-25T16:12:51.758 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-03-25T16:12:54.218 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-03-25T16:12:54.922 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : kernel-modules-core-5.14.0-430.el9.x86_64 1/4 2024-03-25T16:12:55.002 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:12:56.439 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:12:57.712 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:12:57.846 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:13:02.573 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:13:02.705 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : kernel-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:13:07.474 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:13:25.970 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:13:25.971 INFO:teuthology.orchestra.run.smithi097.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-25T16:13:25.971 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T16:13:26.015 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:13:26.390 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: kernel-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:13:26.390 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : kernel-5.14.0-430.el9.x86_64 1/4 2024-03-25T16:13:26.391 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:13:26.391 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:13:26.810 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:13:26.810 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T16:13:26.810 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2024-03-25T16:13:26.810 INFO:teuthology.orchestra.run.smithi097.stdout: kernel-5.14.0-430.el9.x86_64 2024-03-25T16:13:26.810 INFO:teuthology.orchestra.run.smithi097.stdout: kernel-core-5.14.0-430.el9.x86_64 2024-03-25T16:13:26.810 INFO:teuthology.orchestra.run.smithi097.stdout: kernel-modules-5.14.0-430.el9.x86_64 2024-03-25T16:13:26.810 INFO:teuthology.orchestra.run.smithi097.stdout: kernel-modules-core-5.14.0-430.el9.x86_64 2024-03-25T16:13:26.811 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T16:13:26.811 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-03-25T16:13:27.119 DEBUG:teuthology.orchestra.run.smithi097:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T16:13:27.217 INFO:teuthology.orchestra.run.smithi097.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-25T16:13:27.218 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-25T16:13:27.218 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-430.el9.x86_64 2024-03-25T16:13:27.218 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-25T16:13:27.218 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-25T16:13:27.218 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-25T16:13:27.218 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi097.front.sepia.ceph.com, path=None, version=distro) 2024-03-25T16:13:27.219 DEBUG:teuthology.orchestra.run.smithi097:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T16:13:27.299 INFO:teuthology.orchestra.run.smithi097.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-25T16:13:27.300 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-25T16:13:27.300 DEBUG:teuthology.orchestra.run.smithi097:> sudo rpm -qi grub2-tools 2024-03-25T16:13:27.340 INFO:teuthology.orchestra.run.smithi097.stdout:Name : grub2-tools 2024-03-25T16:13:27.340 INFO:teuthology.orchestra.run.smithi097.stdout:Epoch : 1 2024-03-25T16:13:27.340 INFO:teuthology.orchestra.run.smithi097.stdout:Version : 2.06 2024-03-25T16:13:27.340 INFO:teuthology.orchestra.run.smithi097.stdout:Release : 68.el9 2024-03-25T16:13:27.340 INFO:teuthology.orchestra.run.smithi097.stdout:Architecture: x86_64 2024-03-25T16:13:27.340 INFO:teuthology.orchestra.run.smithi097.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-25T16:13:27.340 INFO:teuthology.orchestra.run.smithi097.stdout:Group : Unspecified 2024-03-25T16:13:27.340 INFO:teuthology.orchestra.run.smithi097.stdout:Size : 8264254 2024-03-25T16:13:27.341 INFO:teuthology.orchestra.run.smithi097.stdout:License : GPLv3+ 2024-03-25T16:13:27.341 INFO:teuthology.orchestra.run.smithi097.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-25T16:13:27.341 INFO:teuthology.orchestra.run.smithi097.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-25T16:13:27.341 INFO:teuthology.orchestra.run.smithi097.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-25T16:13:27.341 INFO:teuthology.orchestra.run.smithi097.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-25T16:13:27.341 INFO:teuthology.orchestra.run.smithi097.stdout:Packager : builder@centos.org 2024-03-25T16:13:27.341 INFO:teuthology.orchestra.run.smithi097.stdout:Vendor : CentOS 2024-03-25T16:13:27.341 INFO:teuthology.orchestra.run.smithi097.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-25T16:13:27.341 INFO:teuthology.orchestra.run.smithi097.stdout:Summary : Support tools for GRUB. 2024-03-25T16:13:27.342 INFO:teuthology.orchestra.run.smithi097.stdout:Description : 2024-03-25T16:13:27.342 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T16:13:27.342 INFO:teuthology.orchestra.run.smithi097.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-25T16:13:27.342 INFO:teuthology.orchestra.run.smithi097.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-25T16:13:27.342 INFO:teuthology.orchestra.run.smithi097.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-25T16:13:27.342 INFO:teuthology.orchestra.run.smithi097.stdout:hardware devices. 2024-03-25T16:13:27.342 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T16:13:27.342 INFO:teuthology.orchestra.run.smithi097.stdout:This subpackage provides tools for support of all platforms. 2024-03-25T16:13:27.343 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-25T16:13:27.343 INFO:teuthology.task.kernel:Updating grub on smithi097 to boot 5.14.0-430.el9.x86_64 2024-03-25T16:13:27.343 DEBUG:teuthology.orchestra.run.smithi097:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-25T16:13:27.933 INFO:teuthology.orchestra.run.smithi097.stderr:Generating grub configuration file ... 2024-03-25T16:13:29.313 INFO:teuthology.orchestra.run.smithi097.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-25T16:13:29.350 INFO:teuthology.orchestra.run.smithi097.stderr:done 2024-03-25T16:13:29.352 DEBUG:teuthology.orchestra.run.smithi097:> mktemp 2024-03-25T16:13:29.368 INFO:teuthology.orchestra.run.smithi097.stdout:/tmp/tmp.pR1WZi9083 2024-03-25T16:13:29.368 DEBUG:teuthology.orchestra.run.smithi097:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.pR1WZi9083 2024-03-25T16:13:29.437 DEBUG:teuthology.orchestra.run.smithi097:> sudo chmod 0666 /tmp/tmp.pR1WZi9083 2024-03-25T16:13:29.576 DEBUG:teuthology.orchestra.remote:smithi097:/tmp/tmp.pR1WZi9083 is 6KB 2024-03-25T16:13:29.625 DEBUG:teuthology.orchestra.run.smithi097:> rm -fr /tmp/tmp.pR1WZi9083 2024-03-25T16:13:29.639 DEBUG:teuthology.orchestra.run.smithi097:> sudo /bin/ls /boot/loader/entries || true 2024-03-25T16:13:29.712 INFO:teuthology.orchestra.run.smithi097.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-25T16:13:29.712 INFO:teuthology.orchestra.run.smithi097.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-25T16:13:29.712 INFO:teuthology.orchestra.run.smithi097.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-25T16:13:29.712 INFO:teuthology.orchestra.run.smithi097.stdout:15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64.conf 2024-03-25T16:13:29.713 DEBUG:teuthology.orchestra.run.smithi097:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64 2024-03-25T16:13:29.858 DEBUG:teuthology.orchestra.run.smithi097:> sudo shutdown -r now 2024-03-25T16:13:37.112 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:13:37.112 INFO:teuthology.orchestra.run.smithi098.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-25T16:13:37.112 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T16:13:37.155 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:13:37.538 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: kernel-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:13:37.539 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : kernel-5.14.0-430.el9.x86_64 1/4 2024-03-25T16:13:37.539 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:13:37.539 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:13:37.965 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:13:37.965 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T16:13:37.965 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-03-25T16:13:37.965 INFO:teuthology.orchestra.run.smithi098.stdout: kernel-5.14.0-430.el9.x86_64 2024-03-25T16:13:37.965 INFO:teuthology.orchestra.run.smithi098.stdout: kernel-core-5.14.0-430.el9.x86_64 2024-03-25T16:13:37.965 INFO:teuthology.orchestra.run.smithi098.stdout: kernel-modules-5.14.0-430.el9.x86_64 2024-03-25T16:13:37.965 INFO:teuthology.orchestra.run.smithi098.stdout: kernel-modules-core-5.14.0-430.el9.x86_64 2024-03-25T16:13:37.965 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T16:13:37.965 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-25T16:13:38.329 DEBUG:teuthology.orchestra.run.smithi098:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T16:13:38.439 INFO:teuthology.orchestra.run.smithi098.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-25T16:13:38.440 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-25T16:13:38.440 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-430.el9.x86_64 2024-03-25T16:13:38.440 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-25T16:13:38.440 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-25T16:13:38.440 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-25T16:13:38.440 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi098.front.sepia.ceph.com, path=None, version=distro) 2024-03-25T16:13:38.440 DEBUG:teuthology.orchestra.run.smithi098:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T16:13:38.519 INFO:teuthology.orchestra.run.smithi098.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-25T16:13:38.519 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-25T16:13:38.520 DEBUG:teuthology.orchestra.run.smithi098:> sudo rpm -qi grub2-tools 2024-03-25T16:13:38.553 INFO:teuthology.orchestra.run.smithi098.stdout:Name : grub2-tools 2024-03-25T16:13:38.553 INFO:teuthology.orchestra.run.smithi098.stdout:Epoch : 1 2024-03-25T16:13:38.554 INFO:teuthology.orchestra.run.smithi098.stdout:Version : 2.06 2024-03-25T16:13:38.554 INFO:teuthology.orchestra.run.smithi098.stdout:Release : 68.el9 2024-03-25T16:13:38.554 INFO:teuthology.orchestra.run.smithi098.stdout:Architecture: x86_64 2024-03-25T16:13:38.554 INFO:teuthology.orchestra.run.smithi098.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-25T16:13:38.554 INFO:teuthology.orchestra.run.smithi098.stdout:Group : Unspecified 2024-03-25T16:13:38.554 INFO:teuthology.orchestra.run.smithi098.stdout:Size : 8264254 2024-03-25T16:13:38.554 INFO:teuthology.orchestra.run.smithi098.stdout:License : GPLv3+ 2024-03-25T16:13:38.554 INFO:teuthology.orchestra.run.smithi098.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-25T16:13:38.554 INFO:teuthology.orchestra.run.smithi098.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-25T16:13:38.554 INFO:teuthology.orchestra.run.smithi098.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-25T16:13:38.554 INFO:teuthology.orchestra.run.smithi098.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-25T16:13:38.554 INFO:teuthology.orchestra.run.smithi098.stdout:Packager : builder@centos.org 2024-03-25T16:13:38.554 INFO:teuthology.orchestra.run.smithi098.stdout:Vendor : CentOS 2024-03-25T16:13:38.555 INFO:teuthology.orchestra.run.smithi098.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-25T16:13:38.555 INFO:teuthology.orchestra.run.smithi098.stdout:Summary : Support tools for GRUB. 2024-03-25T16:13:38.555 INFO:teuthology.orchestra.run.smithi098.stdout:Description : 2024-03-25T16:13:38.555 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T16:13:38.555 INFO:teuthology.orchestra.run.smithi098.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-25T16:13:38.555 INFO:teuthology.orchestra.run.smithi098.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-25T16:13:38.555 INFO:teuthology.orchestra.run.smithi098.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-25T16:13:38.555 INFO:teuthology.orchestra.run.smithi098.stdout:hardware devices. 2024-03-25T16:13:38.555 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T16:13:38.555 INFO:teuthology.orchestra.run.smithi098.stdout:This subpackage provides tools for support of all platforms. 2024-03-25T16:13:38.556 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-25T16:13:38.556 INFO:teuthology.task.kernel:Updating grub on smithi098 to boot 5.14.0-430.el9.x86_64 2024-03-25T16:13:38.556 DEBUG:teuthology.orchestra.run.smithi098:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-25T16:13:39.153 INFO:teuthology.orchestra.run.smithi098.stderr:Generating grub configuration file ... 2024-03-25T16:13:40.500 INFO:teuthology.orchestra.run.smithi098.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-25T16:13:40.533 INFO:teuthology.orchestra.run.smithi098.stderr:done 2024-03-25T16:13:40.535 DEBUG:teuthology.orchestra.run.smithi098:> mktemp 2024-03-25T16:13:40.551 INFO:teuthology.orchestra.run.smithi098.stdout:/tmp/tmp.hKd7kHwmt5 2024-03-25T16:13:40.551 DEBUG:teuthology.orchestra.run.smithi098:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.hKd7kHwmt5 2024-03-25T16:13:40.619 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 0666 /tmp/tmp.hKd7kHwmt5 2024-03-25T16:13:40.751 DEBUG:teuthology.orchestra.remote:smithi098:/tmp/tmp.hKd7kHwmt5 is 6KB 2024-03-25T16:13:40.800 DEBUG:teuthology.orchestra.run.smithi098:> rm -fr /tmp/tmp.hKd7kHwmt5 2024-03-25T16:13:40.814 DEBUG:teuthology.orchestra.run.smithi098:> sudo /bin/ls /boot/loader/entries || true 2024-03-25T16:13:40.879 INFO:teuthology.orchestra.run.smithi098.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-25T16:13:40.879 INFO:teuthology.orchestra.run.smithi098.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-25T16:13:40.879 INFO:teuthology.orchestra.run.smithi098.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-25T16:13:40.879 INFO:teuthology.orchestra.run.smithi098.stdout:15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64.conf 2024-03-25T16:13:40.880 DEBUG:teuthology.orchestra.run.smithi098:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64 2024-03-25T16:13:41.005 DEBUG:teuthology.orchestra.run.smithi098:> sudo shutdown -r now 2024-03-25T16:13:59.878 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-25T16:13:59.879 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2024-03-25T16:13:59.879 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi097.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:14:11.018 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-25T16:14:11.018 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi098.front.sepia.ceph.com' 2024-03-25T16:14:11.019 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:14:18.398 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.97 2024-03-25T16:14:27.406 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2024-03-25T16:14:27.407 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi097.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:14:29.406 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.98 2024-03-25T16:14:30.462 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.97 2024-03-25T16:14:38.414 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi098.front.sepia.ceph.com' 2024-03-25T16:14:38.415 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:14:41.470 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.98 2024-03-25T16:14:42.464 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2024-03-25T16:14:42.465 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi097.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:14:53.482 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi098.front.sepia.ceph.com' 2024-03-25T16:14:53.483 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:15:14.285 DEBUG:teuthology.orchestra.run.smithi097:> true 2024-03-25T16:15:15.151 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi097.front.sepia.ceph.com' 2024-03-25T16:15:15.151 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-430.el9.x86_64"... 2024-03-25T16:15:15.151 DEBUG:teuthology.orchestra.run.smithi097:> uname -r 2024-03-25T16:15:15.165 INFO:teuthology.orchestra.run.smithi097.stdout:5.14.0-430.el9.x86_64 2024-03-25T16:15:15.165 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-430.el9.x86_64 vs 5.14.0-430.el9.x86_64 2024-03-25T16:15:15.165 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-25T16:15:15.165 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-25T16:15:16.166 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-25T16:15:16.166 DEBUG:teuthology.orchestra.run.smithi097:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-25T16:15:16.225 INFO:teuthology.orchestra.run.smithi097.stdout:ttyS1 2024-03-25T16:15:16.264 DEBUG:teuthology.parallel:result is None 2024-03-25T16:15:25.468 DEBUG:teuthology.orchestra.run.smithi098:> true 2024-03-25T16:15:25.887 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi098.front.sepia.ceph.com' 2024-03-25T16:15:25.887 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-430.el9.x86_64"... 2024-03-25T16:15:25.887 DEBUG:teuthology.orchestra.run.smithi098:> uname -r 2024-03-25T16:15:25.902 INFO:teuthology.orchestra.run.smithi098.stdout:5.14.0-430.el9.x86_64 2024-03-25T16:15:25.902 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-430.el9.x86_64 vs 5.14.0-430.el9.x86_64 2024-03-25T16:15:25.902 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-25T16:15:25.902 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-25T16:15:26.903 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-25T16:15:26.904 DEBUG:teuthology.orchestra.run.smithi098:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-25T16:15:26.990 INFO:teuthology.orchestra.run.smithi098.stdout:ttyS1 2024-03-25T16:15:27.021 DEBUG:teuthology.parallel:result is None 2024-03-25T16:15:27.022 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-25T16:15:27.033 INFO:teuthology.task.internal:Creating test directory... 2024-03-25T16:15:27.033 DEBUG:teuthology.orchestra.run.smithi097:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-25T16:15:27.036 DEBUG:teuthology.orchestra.run.smithi098:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-25T16:15:27.051 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-25T16:15:27.068 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-25T16:15:27.098 INFO:teuthology.task.internal:Creating archive directory... 2024-03-25T16:15:27.099 DEBUG:teuthology.orchestra.run.smithi097:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-25T16:15:27.101 DEBUG:teuthology.orchestra.run.smithi098:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-25T16:15:27.296 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-25T16:15:27.303 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-25T16:15:27.303 DEBUG:teuthology.orchestra.run.smithi097:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-25T16:15:27.306 DEBUG:teuthology.orchestra.run.smithi098:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-25T16:15:27.337 INFO:teuthology.orchestra.run.smithi098.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T16:15:27.350 INFO:teuthology.orchestra.run.smithi098.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T16:15:27.350 INFO:teuthology.orchestra.run.smithi097.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T16:15:27.365 INFO:teuthology.orchestra.run.smithi097.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T16:15:27.366 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-25T16:15:27.374 INFO:teuthology.task.internal:Configuring sudo... 2024-03-25T16:15:27.374 DEBUG:teuthology.orchestra.run.smithi097:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-25T16:15:27.376 DEBUG:teuthology.orchestra.run.smithi098:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-25T16:15:27.419 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-25T16:15:27.428 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-25T16:15:27.429 DEBUG:teuthology.orchestra.run.smithi097:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-25T16:15:27.447 DEBUG:teuthology.orchestra.run.smithi098:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-25T16:15:27.474 DEBUG:teuthology.orchestra.run.smithi097:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T16:15:27.545 DEBUG:teuthology.orchestra.run.smithi097:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T16:15:27.598 DEBUG:teuthology.orchestra.run.smithi097:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T16:15:27.680 DEBUG:teuthology.orchestra.run.smithi097:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T16:15:27.723 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-25T16:15:27.723 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-25T16:15:27.790 DEBUG:teuthology.orchestra.run.smithi098:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T16:15:27.829 DEBUG:teuthology.orchestra.run.smithi098:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T16:15:27.882 DEBUG:teuthology.orchestra.run.smithi098:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T16:15:27.959 DEBUG:teuthology.orchestra.run.smithi098:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T16:15:28.005 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-25T16:15:28.005 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-25T16:15:28.072 DEBUG:teuthology.orchestra.run.smithi097:> sudo service rsyslog restart 2024-03-25T16:15:28.075 DEBUG:teuthology.orchestra.run.smithi098:> sudo service rsyslog restart 2024-03-25T16:15:28.120 INFO:teuthology.orchestra.run.smithi097.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T16:15:28.194 INFO:teuthology.orchestra.run.smithi098.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T16:15:28.672 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-25T16:15:28.680 INFO:teuthology.task.internal:Starting timer... 2024-03-25T16:15:28.681 INFO:teuthology.run_tasks:Running task pcp... 2024-03-25T16:15:28.703 INFO:teuthology.run_tasks:Running task selinux... 2024-03-25T16:15:28.735 DEBUG:teuthology.orchestra.run.smithi097:> sudo service auditd rotate 2024-03-25T16:15:28.788 INFO:teuthology.orchestra.run.smithi097.stdout:Rotating logs: 2024-03-25T16:15:28.789 DEBUG:teuthology.orchestra.run.smithi098:> sudo service auditd rotate 2024-03-25T16:15:28.870 INFO:teuthology.orchestra.run.smithi098.stdout:Rotating logs: 2024-03-25T16:15:28.874 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-25T16:15:28.874 DEBUG:teuthology.orchestra.run.smithi097:> /usr/sbin/getenforce 2024-03-25T16:15:28.904 INFO:teuthology.orchestra.run.smithi097.stdout:Permissive 2024-03-25T16:15:28.904 DEBUG:teuthology.orchestra.run.smithi098:> /usr/sbin/getenforce 2024-03-25T16:15:28.943 INFO:teuthology.orchestra.run.smithi098.stdout:Permissive 2024-03-25T16:15:28.943 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi097.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi098.front.sepia.ceph.com': 'permissive'} 2024-03-25T16:15:28.943 DEBUG:teuthology.orchestra.run.smithi097:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T16:15:28.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T16:15:28.974 DEBUG:teuthology.orchestra.run.smithi098:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T16:15:29.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T16:15:29.028 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-25T16:15:29.029 DEBUG:teuthology.orchestra.run.smithi097:> sudo /usr/sbin/setenforce permissive 2024-03-25T16:15:29.064 DEBUG:teuthology.orchestra.run.smithi098:> sudo /usr/sbin/setenforce permissive 2024-03-25T16:15:29.137 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-25T16:15:29.147 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-25T16:15:29.165 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-25T16:15:29.167 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi097.front.sepia.ceph.com,smithi098.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-25T16:19:40.507 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi097.front.sepia.ceph.com'), Remote(name='ubuntu@smithi098.front.sepia.ceph.com')] 2024-03-25T16:19:40.508 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2024-03-25T16:19:40.509 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi097.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:19:40.597 DEBUG:teuthology.orchestra.run.smithi097:> true 2024-03-25T16:19:40.672 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi097.front.sepia.ceph.com' 2024-03-25T16:19:40.673 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi098.front.sepia.ceph.com' 2024-03-25T16:19:40.673 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:19:40.761 DEBUG:teuthology.orchestra.run.smithi098:> true 2024-03-25T16:19:40.797 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi098.front.sepia.ceph.com' 2024-03-25T16:19:40.797 INFO:teuthology.run_tasks:Running task clock... 2024-03-25T16:19:40.807 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-25T16:19:40.808 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-25T16:19:40.808 DEBUG:teuthology.orchestra.run.smithi097:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T16:19:40.811 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-25T16:19:40.811 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T16:19:40.839 INFO:teuthology.orchestra.run.smithi097.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-25T16:19:40.855 INFO:teuthology.orchestra.run.smithi097.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-25T16:19:40.869 INFO:teuthology.orchestra.run.smithi098.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-25T16:19:40.884 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: ntpd: command not found 2024-03-25T16:19:40.886 INFO:teuthology.orchestra.run.smithi098.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-25T16:19:40.898 INFO:teuthology.orchestra.run.smithi097.stdout:506 Cannot talk to daemon 2024-03-25T16:19:40.914 INFO:teuthology.orchestra.run.smithi097.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-25T16:19:40.915 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: ntpd: command not found 2024-03-25T16:19:40.929 INFO:teuthology.orchestra.run.smithi098.stdout:506 Cannot talk to daemon 2024-03-25T16:19:40.930 INFO:teuthology.orchestra.run.smithi097.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-25T16:19:40.946 INFO:teuthology.orchestra.run.smithi098.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-25T16:19:40.962 INFO:teuthology.orchestra.run.smithi098.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-25T16:19:40.982 INFO:teuthology.orchestra.run.smithi097.stderr:bash: line 1: ntpq: command not found 2024-03-25T16:19:40.985 INFO:teuthology.orchestra.run.smithi097.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T16:19:40.985 INFO:teuthology.orchestra.run.smithi097.stdout:=============================================================================== 2024-03-25T16:19:40.985 INFO:teuthology.orchestra.run.smithi097.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:19:40.985 INFO:teuthology.orchestra.run.smithi097.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:19:40.985 INFO:teuthology.orchestra.run.smithi097.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:19:40.985 INFO:teuthology.orchestra.run.smithi097.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:19:41.021 INFO:teuthology.orchestra.run.smithi098.stderr:bash: line 1: ntpq: command not found 2024-03-25T16:19:41.023 INFO:teuthology.orchestra.run.smithi098.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T16:19:41.024 INFO:teuthology.orchestra.run.smithi098.stdout:=============================================================================== 2024-03-25T16:19:41.024 INFO:teuthology.orchestra.run.smithi098.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:19:41.024 INFO:teuthology.orchestra.run.smithi098.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:19:41.024 INFO:teuthology.orchestra.run.smithi098.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:19:41.024 INFO:teuthology.orchestra.run.smithi098.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:19:41.025 INFO:teuthology.run_tasks:Running task install... 2024-03-25T16:19:41.034 DEBUG:teuthology.task.install:project ceph 2024-03-25T16:19:41.034 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}} 2024-03-25T16:19:41.034 DEBUG:teuthology.task.install:config {'debuginfo': True, 'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'} 2024-03-25T16:19:41.034 INFO:teuthology.task.install:Using flavor: default 2024-03-25T16:19:41.040 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse', 'ceph-common-dbg', 'ceph-fuse-dbg', 'ceph-mds-dbg', 'ceph-mgr-dbg', 'ceph-mon-dbg', 'ceph-osd-dbg', 'ceph-test-dbg', 'libcephfs2-dbg', 'librados2-dbg', 'libradosstriper1-dbg', 'librbd1-dbg', 'librgw2-dbg', 'radosgw-dbg', 'rbd-fuse-dbg', 'rbd-mirror-dbg', 'rbd-nbd-dbg'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd', 'ceph-base-debuginfo', 'ceph-common-debuginfo', 'ceph-immutable-object-cache-debuginfo', 'ceph-radosgw-debuginfo', 'ceph-test-debuginfo', 'ceph-base-debuginfo', 'ceph-mgr-debuginfo', 'ceph-mds-debuginfo', 'ceph-mon-debuginfo', 'ceph-osd-debuginfo', 'ceph-fuse-debuginfo', 'librados-devel-debuginfo', 'libcephfs2-debuginfo', 'librados2-debuginfo', 'librbd1-debuginfo', 'python3-cephfs-debuginfo', 'python3-rados-debuginfo', 'python3-rbd-debuginfo', 'python3-rgw-debuginfo', 'rbd-fuse-debuginfo', 'rbd-mirror-debuginfo', 'rbd-nbd-debuginfo']} 2024-03-25T16:19:41.041 INFO:teuthology.task.install:extra packages: [] 2024-03-25T16:19:41.041 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-25T16:19:41.041 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T16:19:41.043 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-25T16:19:41.043 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T16:19:41.184 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-25T16:19:41.184 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-25T16:19:41.316 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-25T16:19:41.316 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-25T16:19:41.569 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-25T16:19:41.569 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-25T16:19:41.569 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-25T16:19:41.599 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2024-03-25T16:19:41.599 DEBUG:teuthology.orchestra.run.smithi097:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-25T16:19:41.647 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-25T16:19:41.647 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-25T16:19:41.647 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-25T16:19:41.668 DEBUG:teuthology.orchestra.run.smithi097:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-25T16:19:41.680 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2024-03-25T16:19:41.680 DEBUG:teuthology.orchestra.run.smithi098:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-25T16:19:41.750 DEBUG:teuthology.orchestra.run.smithi097:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-25T16:19:41.751 DEBUG:teuthology.orchestra.run.smithi098:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-25T16:19:41.818 INFO:teuthology.orchestra.run.smithi097.stdout:check_obsoletes = 1 2024-03-25T16:19:41.819 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum clean all 2024-03-25T16:19:41.833 DEBUG:teuthology.orchestra.run.smithi098:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-25T16:19:41.900 INFO:teuthology.orchestra.run.smithi098.stdout:check_obsoletes = 1 2024-03-25T16:19:41.902 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum clean all 2024-03-25T16:19:42.070 INFO:teuthology.orchestra.run.smithi097.stdout:74 files removed 2024-03-25T16:19:42.096 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install ceph-radosgw 2024-03-25T16:19:42.149 INFO:teuthology.orchestra.run.smithi098.stdout:74 files removed 2024-03-25T16:19:42.179 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph-radosgw 2024-03-25T16:19:43.236 INFO:teuthology.orchestra.run.smithi098.stdout:ceph packages for x86_64 108 kB/s | 82 kB 00:00 2024-03-25T16:19:43.705 INFO:teuthology.orchestra.run.smithi097.stdout:ceph packages for x86_64 62 kB/s | 82 kB 00:01 2024-03-25T16:19:44.020 INFO:teuthology.orchestra.run.smithi098.stdout:ceph noarch packages 14 kB/s | 10 kB 00:00 2024-03-25T16:19:44.565 INFO:teuthology.orchestra.run.smithi097.stdout:ceph noarch packages 12 kB/s | 10 kB 00:00 2024-03-25T16:19:44.880 INFO:teuthology.orchestra.run.smithi098.stdout:ceph source packages 2.3 kB/s | 1.9 kB 00:00 2024-03-25T16:19:45.315 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS-9 - AppStream 17 MB/s | 6.7 MB 00:00 2024-03-25T16:19:45.449 INFO:teuthology.orchestra.run.smithi097.stdout:ceph source packages 2.3 kB/s | 1.9 kB 00:00 2024-03-25T16:19:45.867 INFO:teuthology.orchestra.run.smithi097.stdout:CentOS-9 - AppStream 18 MB/s | 6.7 MB 00:00 2024-03-25T16:19:46.715 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS-9 - BaseOS 7.2 MB/s | 2.0 MB 00:00 2024-03-25T16:19:47.493 INFO:teuthology.orchestra.run.smithi097.stdout:CentOS-9 - BaseOS 3.9 MB/s | 2.0 MB 00:00 2024-03-25T16:19:48.669 INFO:teuthology.orchestra.run.smithi097.stdout:CentOS Stream 9 - BaseOS 9.4 MB/s | 8.0 MB 00:00 2024-03-25T16:19:50.385 INFO:teuthology.orchestra.run.smithi097.stdout:CentOS Stream 9 - AppStream 27 MB/s | 19 MB 00:00 2024-03-25T16:19:50.905 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - BaseOS 2.1 MB/s | 8.0 MB 00:03 2024-03-25T16:19:53.032 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - AppStream 17 MB/s | 19 MB 00:01 2024-03-25T16:19:54.554 INFO:teuthology.orchestra.run.smithi097.stdout:CentOS Stream 9 - CRB 10 MB/s | 6.1 MB 00:00 2024-03-25T16:19:56.144 INFO:teuthology.orchestra.run.smithi097.stdout:CentOS Stream 9 - Extras packages 61 kB/s | 16 kB 00:00 2024-03-25T16:19:56.916 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - CRB 13 MB/s | 6.1 MB 00:00 2024-03-25T16:19:59.176 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - Extras packages 14 kB/s | 16 kB 00:01 2024-03-25T16:20:00.502 INFO:teuthology.orchestra.run.smithi098.stdout:Extra Packages for Enterprise Linux 17 MB/s | 21 MB 00:01 2024-03-25T16:20:05.394 INFO:teuthology.orchestra.run.smithi098.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-03-25T16:20:07.848 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-25T16:20:07.852 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T16:20:07.852 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repository Size 2024-03-25T16:20:07.852 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T16:20:07.852 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-03-25T16:20:07.852 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-25T16:20:07.853 INFO:teuthology.orchestra.run.smithi098.stdout:Upgrading: 2024-03-25T16:20:07.853 INFO:teuthology.orchestra.run.smithi098.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-25T16:20:07.853 INFO:teuthology.orchestra.run.smithi098.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T16:20:07.853 INFO:teuthology.orchestra.run.smithi098.stdout:Installing dependencies: 2024-03-25T16:20:07.853 INFO:teuthology.orchestra.run.smithi098.stdout: boost-program-options 2024-03-25T16:20:07.854 INFO:teuthology.orchestra.run.smithi098.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-25T16:20:07.854 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-25T16:20:07.854 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T16:20:07.854 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-25T16:20:07.854 INFO:teuthology.orchestra.run.smithi098.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-25T16:20:07.855 INFO:teuthology.orchestra.run.smithi098.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-25T16:20:07.855 INFO:teuthology.orchestra.run.smithi098.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-25T16:20:07.855 INFO:teuthology.orchestra.run.smithi098.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-25T16:20:07.855 INFO:teuthology.orchestra.run.smithi098.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-25T16:20:07.855 INFO:teuthology.orchestra.run.smithi098.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-25T16:20:07.855 INFO:teuthology.orchestra.run.smithi098.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-25T16:20:07.856 INFO:teuthology.orchestra.run.smithi098.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-25T16:20:07.856 INFO:teuthology.orchestra.run.smithi098.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-25T16:20:07.856 INFO:teuthology.orchestra.run.smithi098.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-25T16:20:07.856 INFO:teuthology.orchestra.run.smithi098.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-25T16:20:07.856 INFO:teuthology.orchestra.run.smithi098.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-25T16:20:07.856 INFO:teuthology.orchestra.run.smithi098.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-25T16:20:07.857 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ceph-argparse 2024-03-25T16:20:07.857 INFO:teuthology.orchestra.run.smithi098.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-25T16:20:07.857 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-25T16:20:07.857 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-25T16:20:07.857 INFO:teuthology.orchestra.run.smithi098.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-25T16:20:07.857 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-25T16:20:07.858 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-25T16:20:07.858 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-25T16:20:07.858 INFO:teuthology.orchestra.run.smithi098.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-25T16:20:07.858 INFO:teuthology.orchestra.run.smithi098.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-25T16:20:07.858 INFO:teuthology.orchestra.run.smithi098.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-25T16:20:07.859 INFO:teuthology.orchestra.run.smithi098.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-25T16:20:07.859 INFO:teuthology.orchestra.run.smithi098.stdout:Installing weak dependencies: 2024-03-25T16:20:07.859 INFO:teuthology.orchestra.run.smithi098.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-25T16:20:07.859 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T16:20:07.859 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-03-25T16:20:07.859 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T16:20:07.860 INFO:teuthology.orchestra.run.smithi098.stdout:Install 30 Packages 2024-03-25T16:20:07.860 INFO:teuthology.orchestra.run.smithi098.stdout:Upgrade 2 Packages 2024-03-25T16:20:07.860 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T16:20:07.861 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 58 M 2024-03-25T16:20:07.861 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-03-25T16:20:19.545 INFO:teuthology.orchestra.run.smithi098.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 475 kB/s | 5.3 MB 00:11 2024-03-25T16:20:19.661 INFO:teuthology.orchestra.run.smithi098.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 216 kB/s | 25 kB 00:00 2024-03-25T16:20:20.928 INFO:teuthology.orchestra.run.smithi098.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 586 kB/s | 742 kB 00:01 2024-03-25T16:20:21.695 INFO:teuthology.orchestra.run.smithi098.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 650 kB/s | 498 kB 00:00 2024-03-25T16:20:23.039 INFO:teuthology.orchestra.run.smithi098.stdout:(5/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9. 664 kB/s | 9.7 MB 00:15 2024-03-25T16:20:23.154 INFO:teuthology.orchestra.run.smithi098.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 395 kB/s | 45 kB 00:00 2024-03-25T16:20:23.371 INFO:teuthology.orchestra.run.smithi098.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 606 kB/s | 131 kB 00:00 2024-03-25T16:20:23.680 INFO:teuthology.orchestra.run.smithi098.stdout:(8/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 525 kB/s | 162 kB 00:00 2024-03-25T16:20:24.172 INFO:teuthology.orchestra.run.smithi098.stdout:(9/32): python3-rados-19.0.0-1439.g5dcb190d.el9 652 kB/s | 320 kB 00:00 2024-03-25T16:20:24.564 INFO:teuthology.orchestra.run.smithi098.stdout:(10/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 760 kB/s | 298 kB 00:00 2024-03-25T16:20:24.772 INFO:teuthology.orchestra.run.smithi098.stdout:(11/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 477 kB/s | 99 kB 00:00 2024-03-25T16:20:24.898 INFO:teuthology.orchestra.run.smithi098.stdout:(12/32): boost-program-options-1.75.0-8.el9.x86 848 kB/s | 106 kB 00:00 2024-03-25T16:20:24.940 INFO:teuthology.orchestra.run.smithi098.stdout:(13/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.7 MB/s | 159 kB 00:00 2024-03-25T16:20:24.973 INFO:teuthology.orchestra.run.smithi098.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-03-25T16:20:25.036 INFO:teuthology.orchestra.run.smithi098.stdout:(15/32): librdkafka-1.6.1-102.el9.x86_64.rpm 10 MB/s | 661 kB 00:00 2024-03-25T16:20:25.074 INFO:teuthology.orchestra.run.smithi098.stdout:(16/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 7.4 MB/s | 282 kB 00:00 2024-03-25T16:20:25.108 INFO:teuthology.orchestra.run.smithi098.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 1.2 MB/s | 41 kB 00:00 2024-03-25T16:20:25.266 INFO:teuthology.orchestra.run.smithi098.stdout:(18/32): qatlib-23.11.0-1.el9.x86_64.rpm 1.3 MB/s | 216 kB 00:00 2024-03-25T16:20:25.317 INFO:teuthology.orchestra.run.smithi098.stdout:(19/32): qatlib-service-23.11.0-1.el9.x86_64.rp 698 kB/s | 35 kB 00:00 2024-03-25T16:20:25.367 INFO:teuthology.orchestra.run.smithi098.stdout:(20/32): qatzip-libs-1.1.2-2.el9.x86_64.rpm 895 kB/s | 44 kB 00:00 2024-03-25T16:20:26.059 INFO:teuthology.orchestra.run.smithi098.stdout:(21/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 446 kB/s | 309 kB 00:00 2024-03-25T16:20:27.392 INFO:teuthology.orchestra.run.smithi098.stdout:(22/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 1.0 MB/s | 20 MB 00:19 2024-03-25T16:20:27.809 INFO:teuthology.orchestra.run.smithi098.stdout:(23/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_ 854 kB/s | 5.1 MB 00:06 2024-03-25T16:20:27.826 INFO:teuthology.orchestra.run.smithi098.stdout:(24/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 60 kB/s | 26 kB 00:00 2024-03-25T16:20:28.318 INFO:teuthology.orchestra.run.smithi098.stdout:(25/32): liboath-2.6.7-2.el9.x86_64.rpm 95 kB/s | 49 kB 00:00 2024-03-25T16:20:28.352 INFO:teuthology.orchestra.run.smithi098.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 128 kB/s | 67 kB 00:00 2024-03-25T16:20:28.960 INFO:teuthology.orchestra.run.smithi098.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 315 kB/s | 192 kB 00:00 2024-03-25T16:20:29.452 INFO:teuthology.orchestra.run.smithi098.stdout:(28/32): libarrow-9.0.0-10.el9.x86_64.rpm 1.3 MB/s | 4.4 MB 00:03 2024-03-25T16:20:29.778 INFO:teuthology.orchestra.run.smithi098.stdout:(29/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 575 kB/s | 838 kB 00:01 2024-03-25T16:20:32.786 INFO:teuthology.orchestra.run.smithi098.stdout:(30/32): thrift-0.15.0-2.el9.x86_64.rpm 423 kB/s | 1.6 MB 00:03 2024-03-25T16:20:34.444 INFO:teuthology.orchestra.run.smithi098.stdout:(31/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 687 kB/s | 3.1 MB 00:04 2024-03-25T16:20:39.494 INFO:teuthology.orchestra.run.smithi098.stdout:(32/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 344 kB/s | 3.4 MB 00:10 2024-03-25T16:20:39.498 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:20:39.498 INFO:teuthology.orchestra.run.smithi098.stdout:Total 1.8 MB/s | 58 MB 00:31 2024-03-25T16:20:39.532 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-03-25T16:20:39.571 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-03-25T16:20:39.572 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-03-25T16:20:40.042 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-03-25T16:20:40.042 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-03-25T16:20:41.552 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-03-25T16:20:41.635 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-25T16:20:41.666 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-25T16:20:41.691 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-25T16:20:41.714 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-25T16:20:41.730 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-25T16:20:41.872 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-25T16:20:41.927 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-25T16:20:42.282 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-25T16:20:42.310 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:20:42.386 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:20:42.475 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-25T16:20:42.512 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T16:20:42.566 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T16:20:42.595 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-25T16:20:42.639 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-25T16:20:42.708 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T16:20:42.731 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T16:20:42.794 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T16:20:42.821 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 14/34 2024-03-25T16:20:42.932 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-25T16:20:42.964 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-25T16:20:43.005 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-25T16:20:43.037 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-25T16:20:43.066 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-25T16:20:43.099 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-25T16:20:43.416 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-25T16:20:43.446 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T16:20:43.535 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T16:20:43.567 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-25T16:20:43.610 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-25T16:20:44.069 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-25T16:20:44.098 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T16:20:44.161 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T16:20:44.229 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-25T16:20:44.259 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-25T16:20:46.448 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:20:46.481 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:20:47.037 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:20:47.068 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T16:20:47.159 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T16:20:47.159 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-25T16:20:47.159 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-25T16:20:47.159 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T16:20:47.202 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:20:56.460 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:20:56.460 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /sys 2024-03-25T16:20:56.460 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /proc 2024-03-25T16:20:56.460 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /mnt 2024-03-25T16:20:56.460 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /var/tmp 2024-03-25T16:20:56.460 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /home 2024-03-25T16:20:56.460 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /root 2024-03-25T16:20:56.461 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /tmp 2024-03-25T16:20:56.461 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T16:20:57.341 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T16:20:57.361 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T16:20:57.362 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:20:57.362 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-25T16:20:57.362 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T16:20:57.362 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T16:20:57.362 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T16:20:57.364 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T16:20:57.451 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T16:20:57.451 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:20:58.591 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:20:58.592 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-25T16:20:58.592 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-25T16:20:58.592 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-25T16:20:58.592 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-25T16:20:58.592 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-25T16:20:58.592 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-25T16:20:58.592 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:20:58.592 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-25T16:20:58.592 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-25T16:20:58.592 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-25T16:20:58.592 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-25T16:20:58.592 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T16:20:58.592 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-25T16:20:58.593 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-25T16:20:58.593 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-25T16:20:58.593 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-25T16:20:58.593 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-25T16:20:58.593 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-25T16:20:58.593 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-25T16:20:58.593 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 20/34 2024-03-25T16:20:58.593 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 21/34 2024-03-25T16:20:58.593 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 22/34 2024-03-25T16:20:58.593 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-25T16:20:58.593 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-25T16:20:58.593 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-25T16:20:58.593 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-25T16:20:58.593 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-25T16:20:58.594 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-25T16:20:58.594 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-25T16:20:58.594 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-25T16:20:58.594 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:20:58.594 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-25T16:20:58.594 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-25T16:20:59.608 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:20:59.608 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T16:20:59.608 INFO:teuthology.orchestra.run.smithi098.stdout:Upgraded: 2024-03-25T16:20:59.608 INFO:teuthology.orchestra.run.smithi098.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:20:59.608 INFO:teuthology.orchestra.run.smithi098.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:20:59.608 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-03-25T16:20:59.608 INFO:teuthology.orchestra.run.smithi098.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-25T16:20:59.609 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:20:59.609 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:20:59.609 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:20:59.609 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:20:59.609 INFO:teuthology.orchestra.run.smithi098.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-25T16:20:59.609 INFO:teuthology.orchestra.run.smithi098.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-25T16:20:59.609 INFO:teuthology.orchestra.run.smithi098.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-25T16:20:59.609 INFO:teuthology.orchestra.run.smithi098.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:20:59.609 INFO:teuthology.orchestra.run.smithi098.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-25T16:20:59.609 INFO:teuthology.orchestra.run.smithi098.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-25T16:20:59.609 INFO:teuthology.orchestra.run.smithi098.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-25T16:20:59.609 INFO:teuthology.orchestra.run.smithi098.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:20:59.609 INFO:teuthology.orchestra.run.smithi098.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-25T16:20:59.609 INFO:teuthology.orchestra.run.smithi098.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:20:59.610 INFO:teuthology.orchestra.run.smithi098.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-25T16:20:59.610 INFO:teuthology.orchestra.run.smithi098.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-25T16:20:59.610 INFO:teuthology.orchestra.run.smithi098.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-25T16:20:59.610 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:20:59.610 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:20:59.610 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:20:59.610 INFO:teuthology.orchestra.run.smithi098.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-25T16:20:59.610 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:20:59.610 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:20:59.610 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:20:59.610 INFO:teuthology.orchestra.run.smithi098.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-25T16:20:59.610 INFO:teuthology.orchestra.run.smithi098.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-25T16:20:59.611 INFO:teuthology.orchestra.run.smithi098.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-25T16:20:59.611 INFO:teuthology.orchestra.run.smithi098.stdout: re2-1:20211101-3.el9.x86_64 2024-03-25T16:20:59.611 INFO:teuthology.orchestra.run.smithi098.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-25T16:20:59.611 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T16:20:59.611 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-25T16:20:59.922 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph-test 2024-03-25T16:21:00.481 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:00:55 ago on Mon 25 Mar 2024 04:20:05 PM UTC. 2024-03-25T16:21:00.633 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-25T16:21:00.633 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T16:21:00.634 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repository Size 2024-03-25T16:21:00.634 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T16:21:00.634 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-03-25T16:21:00.634 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-25T16:21:00.634 INFO:teuthology.orchestra.run.smithi098.stdout:Installing dependencies: 2024-03-25T16:21:00.634 INFO:teuthology.orchestra.run.smithi098.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-25T16:21:00.635 INFO:teuthology.orchestra.run.smithi098.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-25T16:21:00.635 INFO:teuthology.orchestra.run.smithi098.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-25T16:21:00.635 INFO:teuthology.orchestra.run.smithi098.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-25T16:21:00.635 INFO:teuthology.orchestra.run.smithi098.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-25T16:21:00.635 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T16:21:00.635 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-03-25T16:21:00.635 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T16:21:00.636 INFO:teuthology.orchestra.run.smithi098.stdout:Install 6 Packages 2024-03-25T16:21:00.636 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T16:21:00.636 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 48 M 2024-03-25T16:21:00.637 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 204 M 2024-03-25T16:21:00.637 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-03-25T16:21:00.922 INFO:teuthology.orchestra.run.smithi098.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.2 MB/s | 217 kB 00:00 2024-03-25T16:21:00.963 INFO:teuthology.orchestra.run.smithi098.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 7.1 MB/s | 300 kB 00:00 2024-03-25T16:21:00.989 INFO:teuthology.orchestra.run.smithi098.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 2.5 MB/s | 63 kB 00:00 2024-03-25T16:21:01.097 INFO:teuthology.orchestra.run.smithi098.stdout:(4/6): jq-1.6-16.el9.x86_64.rpm 1.7 MB/s | 187 kB 00:00 2024-03-25T16:21:02.548 INFO:teuthology.orchestra.run.smithi098.stdout:(5/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 91 kB/s | 163 kB 00:01 2024-03-25T16:22:01.043 INFO:teuthology.orchestra.run.smithi098.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 803 kB/s | 47 MB 01:00 2024-03-25T16:22:01.045 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:22:01.045 INFO:teuthology.orchestra.run.smithi098.stdout:Total 817 kB/s | 48 MB 01:00 2024-03-25T16:22:01.057 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-03-25T16:22:01.077 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-03-25T16:22:01.077 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-03-25T16:22:01.455 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-03-25T16:22:01.455 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-03-25T16:22:02.050 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-03-25T16:22:02.126 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-25T16:22:02.168 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-25T16:22:02.209 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T16:22:02.258 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-25T16:22:02.279 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T16:22:06.720 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T16:22:06.795 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T16:22:09.205 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T16:22:09.205 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-25T16:22:09.205 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-25T16:22:09.205 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T16:22:09.205 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-25T16:22:09.205 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-25T16:22:09.626 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-25T16:22:09.626 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T16:22:09.626 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-03-25T16:22:09.626 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:22:09.626 INFO:teuthology.orchestra.run.smithi098.stdout: jq-1.6-16.el9.x86_64 2024-03-25T16:22:09.626 INFO:teuthology.orchestra.run.smithi098.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:22:09.626 INFO:teuthology.orchestra.run.smithi098.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-25T16:22:09.626 INFO:teuthology.orchestra.run.smithi098.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-25T16:22:09.626 INFO:teuthology.orchestra.run.smithi098.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-25T16:22:09.627 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T16:22:09.627 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-25T16:22:09.947 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph 2024-03-25T16:22:10.513 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:02:05 ago on Mon 25 Mar 2024 04:20:05 PM UTC. 2024-03-25T16:22:10.663 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-25T16:22:10.666 INFO:teuthology.orchestra.run.smithi098.stdout:========================================================================================= 2024-03-25T16:22:10.666 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repository Size 2024-03-25T16:22:10.666 INFO:teuthology.orchestra.run.smithi098.stdout:========================================================================================= 2024-03-25T16:22:10.666 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-03-25T16:22:10.666 INFO:teuthology.orchestra.run.smithi098.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-25T16:22:10.666 INFO:teuthology.orchestra.run.smithi098.stdout:Installing dependencies: 2024-03-25T16:22:10.667 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-25T16:22:10.667 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-25T16:22:10.667 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-25T16:22:10.667 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-25T16:22:10.667 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-25T16:22:10.667 INFO:teuthology.orchestra.run.smithi098.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-25T16:22:10.668 INFO:teuthology.orchestra.run.smithi098.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-25T16:22:10.668 INFO:teuthology.orchestra.run.smithi098.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-25T16:22:10.668 INFO:teuthology.orchestra.run.smithi098.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-25T16:22:10.668 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-25T16:22:10.668 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-25T16:22:10.668 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-25T16:22:10.669 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-25T16:22:10.669 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-25T16:22:10.669 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-25T16:22:10.669 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-25T16:22:10.669 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-25T16:22:10.669 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-25T16:22:10.670 INFO:teuthology.orchestra.run.smithi098.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-25T16:22:10.670 INFO:teuthology.orchestra.run.smithi098.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-25T16:22:10.670 INFO:teuthology.orchestra.run.smithi098.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-25T16:22:10.670 INFO:teuthology.orchestra.run.smithi098.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-25T16:22:10.670 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-25T16:22:10.670 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-25T16:22:10.671 INFO:teuthology.orchestra.run.smithi098.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-25T16:22:10.671 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-25T16:22:10.671 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-25T16:22:10.671 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-25T16:22:10.671 INFO:teuthology.orchestra.run.smithi098.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-25T16:22:10.671 INFO:teuthology.orchestra.run.smithi098.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-25T16:22:10.672 INFO:teuthology.orchestra.run.smithi098.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-25T16:22:10.672 INFO:teuthology.orchestra.run.smithi098.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-25T16:22:10.672 INFO:teuthology.orchestra.run.smithi098.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-25T16:22:10.672 INFO:teuthology.orchestra.run.smithi098.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-25T16:22:10.672 INFO:teuthology.orchestra.run.smithi098.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-25T16:22:10.672 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T16:22:10.673 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-03-25T16:22:10.673 INFO:teuthology.orchestra.run.smithi098.stdout:========================================================================================= 2024-03-25T16:22:10.673 INFO:teuthology.orchestra.run.smithi098.stdout:Install 36 Packages 2024-03-25T16:22:10.673 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T16:22:10.674 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 30 M 2024-03-25T16:22:10.674 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 107 M 2024-03-25T16:22:10.675 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-03-25T16:22:11.346 INFO:teuthology.orchestra.run.smithi098.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 14 kB/s | 6.6 kB 00:00 2024-03-25T16:22:13.505 INFO:teuthology.orchestra.run.smithi098.stdout:(2/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 580 kB/s | 1.5 MB 00:02 2024-03-25T16:22:16.605 INFO:teuthology.orchestra.run.smithi098.stdout:(3/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 379 kB/s | 2.1 MB 00:05 2024-03-25T16:22:17.280 INFO:teuthology.orchestra.run.smithi098.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 367 kB/s | 247 kB 00:00 2024-03-25T16:22:17.397 INFO:teuthology.orchestra.run.smithi098.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.6 MB/s | 187 kB 00:00 2024-03-25T16:22:17.439 INFO:teuthology.orchestra.run.smithi098.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-25T16:22:17.472 INFO:teuthology.orchestra.run.smithi098.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 951 kB/s | 32 kB 00:00 2024-03-25T16:22:17.506 INFO:teuthology.orchestra.run.smithi098.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-03-25T16:22:17.531 INFO:teuthology.orchestra.run.smithi098.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.5 MB/s | 39 kB 00:00 2024-03-25T16:22:17.573 INFO:teuthology.orchestra.run.smithi098.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.7 MB/s | 241 kB 00:00 2024-03-25T16:22:17.640 INFO:teuthology.orchestra.run.smithi098.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 17 MB/s | 1.1 MB 00:00 2024-03-25T16:22:17.674 INFO:teuthology.orchestra.run.smithi098.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 3.0 MB/s | 103 kB 00:00 2024-03-25T16:22:17.707 INFO:teuthology.orchestra.run.smithi098.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 3.6 MB/s | 124 kB 00:00 2024-03-25T16:22:17.749 INFO:teuthology.orchestra.run.smithi098.stdout:(14/36): python3-urllib3-1.26.5-5.el9.noarch.rp 4.4 MB/s | 187 kB 00:00 2024-03-25T16:22:18.550 INFO:teuthology.orchestra.run.smithi098.stdout:(15/36): python3-requests-2.25.1-8.el9.noarch.r 156 kB/s | 125 kB 00:00 2024-03-25T16:22:18.625 INFO:teuthology.orchestra.run.smithi098.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 298 kB/s | 22 kB 00:00 2024-03-25T16:22:18.859 INFO:teuthology.orchestra.run.smithi098.stdout:(17/36): luarocks-3.9.2-1.el9.noarch.rpm 648 kB/s | 151 kB 00:00 2024-03-25T16:22:18.917 INFO:teuthology.orchestra.run.smithi098.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 741 kB/s | 43 kB 00:00 2024-03-25T16:22:19.026 INFO:teuthology.orchestra.run.smithi098.stdout:(19/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.5 MB/s | 172 kB 00:00 2024-03-25T16:22:19.143 INFO:teuthology.orchestra.run.smithi098.stdout:(20/36): python3-cherrypy-18.6.1-2.el9.noarch.r 3.0 MB/s | 358 kB 00:00 2024-03-25T16:22:19.210 INFO:teuthology.orchestra.run.smithi098.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 160 kB/s | 11 kB 00:00 2024-03-25T16:22:19.269 INFO:teuthology.orchestra.run.smithi098.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 303 kB/s | 18 kB 00:00 2024-03-25T16:22:19.327 INFO:teuthology.orchestra.run.smithi098.stdout:(23/36): python3-jaraco-collections-3.0.0-8.el9 398 kB/s | 23 kB 00:00 2024-03-25T16:22:19.394 INFO:teuthology.orchestra.run.smithi098.stdout:(24/36): python3-jaraco-functools-3.5.0-2.el9.n 292 kB/s | 19 kB 00:00 2024-03-25T16:22:19.461 INFO:teuthology.orchestra.run.smithi098.stdout:(25/36): python3-jaraco-text-3.2.0-6.el9.noarch 295 kB/s | 20 kB 00:00 2024-03-25T16:22:19.528 INFO:teuthology.orchestra.run.smithi098.stdout:(26/36): python3-logutils-0.3.5-21.el9.noarch.r 696 kB/s | 46 kB 00:00 2024-03-25T16:22:19.603 INFO:teuthology.orchestra.run.smithi098.stdout:(27/36): python3-more-itertools-8.12.0-2.el9.no 1.0 MB/s | 79 kB 00:00 2024-03-25T16:22:19.678 INFO:teuthology.orchestra.run.smithi098.stdout:(28/36): python3-pecan-1.4.2-3.el9.noarch.rpm 3.5 MB/s | 272 kB 00:00 2024-03-25T16:22:19.745 INFO:teuthology.orchestra.run.smithi098.stdout:(29/36): python3-portend-3.1.0-2.el9.noarch.rpm 246 kB/s | 16 kB 00:00 2024-03-25T16:22:19.862 INFO:teuthology.orchestra.run.smithi098.stdout:(30/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86 563 kB/s | 4.7 MB 00:08 2024-03-25T16:22:19.888 INFO:teuthology.orchestra.run.smithi098.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 633 kB/s | 90 kB 00:00 2024-03-25T16:22:19.963 INFO:teuthology.orchestra.run.smithi098.stdout:(32/36): python3-webob-1.8.7-6.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-03-25T16:22:20.013 INFO:teuthology.orchestra.run.smithi098.stdout:(33/36): python3-tempora-5.0.0-2.el9.noarch.rpm 238 kB/s | 36 kB 00:00 2024-03-25T16:22:20.038 INFO:teuthology.orchestra.run.smithi098.stdout:(34/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 5.5 MB/s | 427 kB 00:00 2024-03-25T16:22:20.089 INFO:teuthology.orchestra.run.smithi098.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 265 kB/s | 20 kB 00:00 2024-03-25T16:22:21.865 INFO:teuthology.orchestra.run.smithi098.stdout:(36/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 2.0 MB/s | 17 MB 00:08 2024-03-25T16:22:21.868 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:22:21.868 INFO:teuthology.orchestra.run.smithi098.stdout:Total 2.6 MB/s | 30 MB 00:11 2024-03-25T16:22:21.892 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-03-25T16:22:21.934 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-03-25T16:22:21.934 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-03-25T16:22:22.335 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-03-25T16:22:22.335 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-03-25T16:22:23.642 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-03-25T16:22:23.691 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-25T16:22:23.742 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-25T16:22:23.777 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-25T16:22:23.851 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-25T16:22:23.893 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-25T16:22:23.926 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-25T16:22:24.027 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-25T16:22:24.105 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-25T16:22:24.146 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-25T16:22:24.217 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-25T16:22:24.271 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-25T16:22:24.310 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-25T16:22:24.359 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-25T16:22:24.422 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-25T16:22:24.608 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-25T16:22:24.656 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-25T16:22:24.694 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-25T16:22:24.729 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-25T16:22:24.759 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-25T16:22:24.836 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-25T16:22:24.869 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T16:22:24.902 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-25T16:22:24.942 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-25T16:22:24.978 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-25T16:22:25.012 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-25T16:22:25.102 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-25T16:22:25.143 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-25T16:22:25.215 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-25T16:22:25.353 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-25T16:22:25.478 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T16:22:25.645 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-25T16:22:25.674 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T16:22:25.698 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T16:22:25.698 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:22:25.699 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-25T16:22:25.699 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T16:22:25.699 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T16:22:25.699 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T16:22:27.133 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T16:22:27.159 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T16:22:27.159 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:22:27.159 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-25T16:22:27.159 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T16:22:27.160 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T16:22:27.160 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T16:22:27.601 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T16:22:27.623 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T16:22:27.624 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:22:27.624 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-25T16:22:27.624 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T16:22:27.624 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T16:22:27.624 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T16:22:27.843 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T16:22:27.866 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T16:22:27.866 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:22:27.866 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-25T16:22:27.866 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T16:22:27.866 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T16:22:27.866 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T16:22:27.932 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T16:22:29.790 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T16:22:29.790 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-25T16:22:29.790 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-25T16:22:29.790 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-25T16:22:29.790 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-25T16:22:29.790 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-25T16:22:29.791 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-25T16:22:29.791 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-25T16:22:29.791 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-25T16:22:29.791 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-25T16:22:29.791 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-25T16:22:29.791 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-25T16:22:29.791 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-25T16:22:29.791 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-25T16:22:29.791 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-25T16:22:29.791 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-25T16:22:29.791 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-25T16:22:29.791 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-25T16:22:29.791 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-25T16:22:29.791 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-25T16:22:29.792 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-25T16:22:29.792 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T16:22:29.792 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-25T16:22:29.792 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-25T16:22:29.792 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-25T16:22:29.792 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-25T16:22:29.792 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-25T16:22:29.793 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-25T16:22:29.793 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-25T16:22:29.793 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-25T16:22:29.793 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T16:22:29.793 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-25T16:22:29.793 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-25T16:22:29.794 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-25T16:22:29.794 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-25T16:22:29.794 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-25T16:22:30.839 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-25T16:22:30.839 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T16:22:30.839 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-03-25T16:22:30.839 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:22:30.839 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:22:30.839 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:22:30.839 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:22:30.839 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:22:30.840 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:22:30.840 INFO:teuthology.orchestra.run.smithi098.stdout: lua-5.4.4-4.el9.x86_64 2024-03-25T16:22:30.840 INFO:teuthology.orchestra.run.smithi098.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-25T16:22:30.840 INFO:teuthology.orchestra.run.smithi098.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-25T16:22:30.840 INFO:teuthology.orchestra.run.smithi098.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-25T16:22:30.840 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-25T16:22:30.840 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-25T16:22:30.840 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-25T16:22:30.840 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-25T16:22:30.840 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-25T16:22:30.840 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-25T16:22:30.840 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-25T16:22:30.840 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-25T16:22:30.841 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-25T16:22:30.841 INFO:teuthology.orchestra.run.smithi098.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-25T16:22:30.841 INFO:teuthology.orchestra.run.smithi098.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-25T16:22:30.841 INFO:teuthology.orchestra.run.smithi098.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-25T16:22:30.841 INFO:teuthology.orchestra.run.smithi098.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-25T16:22:30.841 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-25T16:22:30.841 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ply-3.11-14.el9.noarch 2024-03-25T16:22:30.841 INFO:teuthology.orchestra.run.smithi098.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-25T16:22:30.841 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-25T16:22:30.841 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-25T16:22:30.841 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-25T16:22:30.841 INFO:teuthology.orchestra.run.smithi098.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-25T16:22:30.841 INFO:teuthology.orchestra.run.smithi098.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-25T16:22:30.841 INFO:teuthology.orchestra.run.smithi098.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-25T16:22:30.842 INFO:teuthology.orchestra.run.smithi098.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-25T16:22:30.842 INFO:teuthology.orchestra.run.smithi098.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-25T16:22:30.842 INFO:teuthology.orchestra.run.smithi098.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-25T16:22:30.842 INFO:teuthology.orchestra.run.smithi098.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-25T16:22:30.842 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T16:22:30.842 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-25T16:22:31.167 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph-base 2024-03-25T16:22:31.731 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:02:26 ago on Mon 25 Mar 2024 04:20:05 PM UTC. 2024-03-25T16:22:31.830 INFO:teuthology.orchestra.run.smithi098.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:22:31.881 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-25T16:22:31.882 INFO:teuthology.orchestra.run.smithi098.stdout:Nothing to do. 2024-03-25T16:22:31.882 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-25T16:22:31.950 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install cephadm 2024-03-25T16:22:32.505 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:02:27 ago on Mon 25 Mar 2024 04:20:05 PM UTC. 2024-03-25T16:22:32.653 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-25T16:22:32.654 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T16:22:32.654 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repository Size 2024-03-25T16:22:32.655 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T16:22:32.655 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-03-25T16:22:32.655 INFO:teuthology.orchestra.run.smithi098.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-25T16:22:32.655 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T16:22:32.655 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-03-25T16:22:32.655 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T16:22:32.655 INFO:teuthology.orchestra.run.smithi098.stdout:Install 1 Package 2024-03-25T16:22:32.655 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T16:22:32.655 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 747 k 2024-03-25T16:22:32.655 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 754 k 2024-03-25T16:22:32.655 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-03-25T16:22:33.997 INFO:teuthology.orchestra.run.smithi098.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 557 kB/s | 747 kB 00:01 2024-03-25T16:22:33.997 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:22:33.998 INFO:teuthology.orchestra.run.smithi098.stdout:Total 557 kB/s | 747 kB 00:01 2024-03-25T16:22:33.998 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-03-25T16:22:34.004 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-03-25T16:22:34.004 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-03-25T16:22:34.010 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-03-25T16:22:34.010 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-03-25T16:22:34.087 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-03-25T16:22:34.517 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:22:34.610 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:22:34.979 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:22:35.293 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:22:35.293 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T16:22:35.293 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-03-25T16:22:35.293 INFO:teuthology.orchestra.run.smithi098.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:22:35.293 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T16:22:35.293 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-25T16:22:35.491 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph-immutable-object-cache 2024-03-25T16:22:36.068 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:02:31 ago on Mon 25 Mar 2024 04:20:05 PM UTC. 2024-03-25T16:22:36.223 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-25T16:22:36.224 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T16:22:36.224 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repo Size 2024-03-25T16:22:36.224 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T16:22:36.224 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-03-25T16:22:36.224 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-25T16:22:36.224 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T16:22:36.225 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-03-25T16:22:36.225 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T16:22:36.225 INFO:teuthology.orchestra.run.smithi098.stdout:Install 1 Package 2024-03-25T16:22:36.225 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T16:22:36.225 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 145 k 2024-03-25T16:22:36.225 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 436 k 2024-03-25T16:22:36.225 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-03-25T16:22:36.945 INFO:teuthology.orchestra.run.smithi098.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 202 kB/s | 145 kB 00:00 2024-03-25T16:22:36.945 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:22:36.946 INFO:teuthology.orchestra.run.smithi098.stdout:Total 201 kB/s | 145 kB 00:00 2024-03-25T16:22:36.946 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-03-25T16:22:36.954 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-03-25T16:22:36.954 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-03-25T16:22:37.009 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-03-25T16:22:37.009 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-03-25T16:22:37.146 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-03-25T16:22:37.204 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:22:37.228 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:22:37.228 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:22:37.228 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-25T16:22:37.228 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T16:22:38.222 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:22:38.222 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T16:22:38.222 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-03-25T16:22:38.222 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:22:38.222 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T16:22:38.222 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-25T16:22:38.482 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph-mgr 2024-03-25T16:22:39.052 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:02:34 ago on Mon 25 Mar 2024 04:20:05 PM UTC. 2024-03-25T16:22:39.154 INFO:teuthology.orchestra.run.smithi098.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:22:39.208 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-25T16:22:39.209 INFO:teuthology.orchestra.run.smithi098.stdout:Nothing to do. 2024-03-25T16:22:39.209 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-25T16:22:39.279 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph-mgr-dashboard 2024-03-25T16:22:39.868 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:02:34 ago on Mon 25 Mar 2024 04:20:05 PM UTC. 2024-03-25T16:22:40.023 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-25T16:22:40.024 INFO:teuthology.orchestra.run.smithi098.stdout:====================================================================================== 2024-03-25T16:22:40.024 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repository Size 2024-03-25T16:22:40.024 INFO:teuthology.orchestra.run.smithi098.stdout:====================================================================================== 2024-03-25T16:22:40.024 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-03-25T16:22:40.024 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-25T16:22:40.024 INFO:teuthology.orchestra.run.smithi098.stdout:Installing dependencies: 2024-03-25T16:22:40.024 INFO:teuthology.orchestra.run.smithi098.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-25T16:22:40.025 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-25T16:22:40.025 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-25T16:22:40.025 INFO:teuthology.orchestra.run.smithi098.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-25T16:22:40.025 INFO:teuthology.orchestra.run.smithi098.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-25T16:22:40.025 INFO:teuthology.orchestra.run.smithi098.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-25T16:22:40.025 INFO:teuthology.orchestra.run.smithi098.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-25T16:22:40.025 INFO:teuthology.orchestra.run.smithi098.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-25T16:22:40.025 INFO:teuthology.orchestra.run.smithi098.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-25T16:22:40.025 INFO:teuthology.orchestra.run.smithi098.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-25T16:22:40.025 INFO:teuthology.orchestra.run.smithi098.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-25T16:22:40.025 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T16:22:40.025 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-03-25T16:22:40.025 INFO:teuthology.orchestra.run.smithi098.stdout:====================================================================================== 2024-03-25T16:22:40.025 INFO:teuthology.orchestra.run.smithi098.stdout:Install 12 Packages 2024-03-25T16:22:40.025 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T16:22:40.026 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 6.9 M 2024-03-25T16:22:40.026 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 26 M 2024-03-25T16:22:40.026 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-03-25T16:22:40.616 INFO:teuthology.orchestra.run.smithi098.stdout:(1/12): ceph-grafana-dashboards-19.0.0-1439.g5d 68 kB/s | 31 kB 00:00 2024-03-25T16:22:40.708 INFO:teuthology.orchestra.run.smithi098.stdout:(2/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 28 kB/s | 15 kB 00:00 2024-03-25T16:22:40.809 INFO:teuthology.orchestra.run.smithi098.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 2.4 MB/s | 238 kB 00:00 2024-03-25T16:22:40.842 INFO:teuthology.orchestra.run.smithi098.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 4.5 MB/s | 1.0 MB 00:00 2024-03-25T16:22:41.109 INFO:teuthology.orchestra.run.smithi098.stdout:(5/12): protobuf-compiler-3.14.0-13.el9.x86_64. 2.8 MB/s | 863 kB 00:00 2024-03-25T16:22:41.160 INFO:teuthology.orchestra.run.smithi098.stdout:(6/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.7 MB/s | 548 kB 00:00 2024-03-25T16:22:41.277 INFO:teuthology.orchestra.run.smithi098.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 116 kB/s | 19 kB 00:00 2024-03-25T16:22:41.610 INFO:teuthology.orchestra.run.smithi098.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 433 kB/s | 144 kB 00:00 2024-03-25T16:22:41.727 INFO:teuthology.orchestra.run.smithi098.stdout:(9/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 3.6 MB/s | 2.0 MB 00:00 2024-03-25T16:22:41.844 INFO:teuthology.orchestra.run.smithi098.stdout:(10/12): python3-repoze-lru-0.7-16.el9.noarch.r 132 kB/s | 31 kB 00:00 2024-03-25T16:22:42.036 INFO:teuthology.orchestra.run.smithi098.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 611 kB/s | 188 kB 00:00 2024-03-25T16:22:42.728 INFO:teuthology.orchestra.run.smithi098.stdout:(12/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb19 710 kB/s | 1.8 MB 00:02 2024-03-25T16:22:42.733 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:22:42.733 INFO:teuthology.orchestra.run.smithi098.stdout:Total 2.5 MB/s | 6.9 MB 00:02 2024-03-25T16:22:42.752 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-03-25T16:22:42.771 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-03-25T16:22:42.771 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-03-25T16:22:42.946 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-03-25T16:22:42.947 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-03-25T16:22:43.469 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-03-25T16:22:43.529 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-25T16:22:43.563 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-25T16:22:43.610 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-25T16:22:43.641 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-25T16:22:43.754 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-25T16:22:43.890 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-25T16:22:44.007 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-25T16:22:44.062 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-25T16:22:44.091 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-25T16:22:44.123 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-25T16:22:44.364 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-25T16:22:44.398 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T16:22:45.195 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T16:22:45.195 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-25T16:22:45.196 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-25T16:22:45.196 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-25T16:22:45.196 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-25T16:22:45.196 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-25T16:22:45.196 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-25T16:22:45.196 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-25T16:22:45.196 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-25T16:22:45.196 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-25T16:22:45.196 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-25T16:22:45.196 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-25T16:22:45.552 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-25T16:22:45.552 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T16:22:45.552 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-03-25T16:22:45.552 INFO:teuthology.orchestra.run.smithi098.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-25T16:22:45.553 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:22:45.553 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:22:45.553 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:22:45.553 INFO:teuthology.orchestra.run.smithi098.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-25T16:22:45.553 INFO:teuthology.orchestra.run.smithi098.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-25T16:22:45.553 INFO:teuthology.orchestra.run.smithi098.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-25T16:22:45.553 INFO:teuthology.orchestra.run.smithi098.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-25T16:22:45.553 INFO:teuthology.orchestra.run.smithi098.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-25T16:22:45.553 INFO:teuthology.orchestra.run.smithi098.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-25T16:22:45.553 INFO:teuthology.orchestra.run.smithi098.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-25T16:22:45.553 INFO:teuthology.orchestra.run.smithi098.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-25T16:22:45.553 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T16:22:45.553 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-25T16:22:45.814 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-25T16:22:46.373 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:02:41 ago on Mon 25 Mar 2024 04:20:05 PM UTC. 2024-03-25T16:22:46.523 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-25T16:22:46.524 INFO:teuthology.orchestra.run.smithi098.stdout:============================================================================================ 2024-03-25T16:22:46.524 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repository Size 2024-03-25T16:22:46.524 INFO:teuthology.orchestra.run.smithi098.stdout:============================================================================================ 2024-03-25T16:22:46.524 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-03-25T16:22:46.524 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-25T16:22:46.524 INFO:teuthology.orchestra.run.smithi098.stdout:Installing dependencies: 2024-03-25T16:22:46.524 INFO:teuthology.orchestra.run.smithi098.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-25T16:22:46.525 INFO:teuthology.orchestra.run.smithi098.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-25T16:22:46.525 INFO:teuthology.orchestra.run.smithi098.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-25T16:22:46.525 INFO:teuthology.orchestra.run.smithi098.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-25T16:22:46.525 INFO:teuthology.orchestra.run.smithi098.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-25T16:22:46.525 INFO:teuthology.orchestra.run.smithi098.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-25T16:22:46.525 INFO:teuthology.orchestra.run.smithi098.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-25T16:22:46.525 INFO:teuthology.orchestra.run.smithi098.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-25T16:22:46.526 INFO:teuthology.orchestra.run.smithi098.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-25T16:22:46.526 INFO:teuthology.orchestra.run.smithi098.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-25T16:22:46.526 INFO:teuthology.orchestra.run.smithi098.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-25T16:22:46.526 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T16:22:46.526 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-03-25T16:22:46.526 INFO:teuthology.orchestra.run.smithi098.stdout:============================================================================================ 2024-03-25T16:22:46.526 INFO:teuthology.orchestra.run.smithi098.stdout:Install 12 Packages 2024-03-25T16:22:46.526 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T16:22:46.527 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 37 M 2024-03-25T16:22:46.527 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 209 M 2024-03-25T16:22:46.527 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-03-25T16:22:46.952 INFO:teuthology.orchestra.run.smithi098.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 274 kB/s | 32 kB 00:00 2024-03-25T16:22:47.002 INFO:teuthology.orchestra.run.smithi098.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 639 kB/s | 17 kB 00:00 2024-03-25T16:22:47.027 INFO:teuthology.orchestra.run.smithi098.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.3 MB/s | 34 kB 00:00 2024-03-25T16:22:47.228 INFO:teuthology.orchestra.run.smithi098.stdout:(4/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 24 MB/s | 4.7 MB 00:00 2024-03-25T16:22:47.362 INFO:teuthology.orchestra.run.smithi098.stdout:(5/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 36 MB/s | 4.8 MB 00:00 2024-03-25T16:22:47.420 INFO:teuthology.orchestra.run.smithi098.stdout:(6/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 5.1 MB/s | 3.0 MB 00:00 2024-03-25T16:22:47.446 INFO:teuthology.orchestra.run.smithi098.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.6 MB/s | 217 kB 00:00 2024-03-25T16:22:47.913 INFO:teuthology.orchestra.run.smithi098.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 32 MB/s | 16 MB 00:00 2024-03-25T16:22:48.054 INFO:teuthology.orchestra.run.smithi098.stdout:(9/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.3 MB/s | 803 kB 00:00 2024-03-25T16:22:48.080 INFO:teuthology.orchestra.run.smithi098.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 1.1 MB/s | 195 kB 00:00 2024-03-25T16:22:48.305 INFO:teuthology.orchestra.run.smithi098.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 981 kB/s | 245 kB 00:00 2024-03-25T16:23:01.121 INFO:teuthology.orchestra.run.smithi098.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 530 kB/s | 7.4 MB 00:14 2024-03-25T16:23:01.123 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:23:01.123 INFO:teuthology.orchestra.run.smithi098.stdout:Total 2.5 MB/s | 37 MB 00:14 2024-03-25T16:23:01.166 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-03-25T16:23:01.191 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-03-25T16:23:01.191 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-03-25T16:23:01.548 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-03-25T16:23:01.548 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-03-25T16:23:02.251 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-03-25T16:23:02.327 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-25T16:23:02.366 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-25T16:23:02.459 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-25T16:23:02.503 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-25T16:23:02.681 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T16:23:02.719 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T16:23:02.804 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-25T16:23:03.441 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-25T16:23:03.556 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-25T16:23:04.784 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-25T16:23:05.798 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-25T16:23:05.828 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T16:23:08.466 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T16:23:08.467 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-25T16:23:08.467 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-25T16:23:08.467 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-25T16:23:08.467 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-25T16:23:08.467 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T16:23:08.467 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T16:23:08.467 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-25T16:23:08.467 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-25T16:23:08.467 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-25T16:23:08.467 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-25T16:23:08.467 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-25T16:23:08.879 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-25T16:23:08.879 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T16:23:08.879 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-03-25T16:23:08.880 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:23:08.880 INFO:teuthology.orchestra.run.smithi098.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-25T16:23:08.880 INFO:teuthology.orchestra.run.smithi098.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-25T16:23:08.880 INFO:teuthology.orchestra.run.smithi098.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-25T16:23:08.880 INFO:teuthology.orchestra.run.smithi098.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-25T16:23:08.880 INFO:teuthology.orchestra.run.smithi098.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-25T16:23:08.880 INFO:teuthology.orchestra.run.smithi098.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-25T16:23:08.880 INFO:teuthology.orchestra.run.smithi098.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-25T16:23:08.880 INFO:teuthology.orchestra.run.smithi098.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-25T16:23:08.880 INFO:teuthology.orchestra.run.smithi098.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-25T16:23:08.880 INFO:teuthology.orchestra.run.smithi098.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-25T16:23:08.880 INFO:teuthology.orchestra.run.smithi098.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-25T16:23:08.880 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T16:23:08.880 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-25T16:23:09.249 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph-mgr-rook 2024-03-25T16:23:09.841 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:03:04 ago on Mon 25 Mar 2024 04:20:05 PM UTC. 2024-03-25T16:23:10.003 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-25T16:23:10.004 INFO:teuthology.orchestra.run.smithi098.stdout:======================================================================================== 2024-03-25T16:23:10.004 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repository Size 2024-03-25T16:23:10.004 INFO:teuthology.orchestra.run.smithi098.stdout:======================================================================================== 2024-03-25T16:23:10.004 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-03-25T16:23:10.004 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-25T16:23:10.004 INFO:teuthology.orchestra.run.smithi098.stdout:Installing dependencies: 2024-03-25T16:23:10.004 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-25T16:23:10.004 INFO:teuthology.orchestra.run.smithi098.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-25T16:23:10.004 INFO:teuthology.orchestra.run.smithi098.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-25T16:23:10.004 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-25T16:23:10.004 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-25T16:23:10.005 INFO:teuthology.orchestra.run.smithi098.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-25T16:23:10.005 INFO:teuthology.orchestra.run.smithi098.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-25T16:23:10.005 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-25T16:23:10.005 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-25T16:23:10.005 INFO:teuthology.orchestra.run.smithi098.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-25T16:23:10.005 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-25T16:23:10.005 INFO:teuthology.orchestra.run.smithi098.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-25T16:23:10.005 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T16:23:10.005 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-03-25T16:23:10.005 INFO:teuthology.orchestra.run.smithi098.stdout:======================================================================================== 2024-03-25T16:23:10.005 INFO:teuthology.orchestra.run.smithi098.stdout:Install 13 Packages 2024-03-25T16:23:10.005 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T16:23:10.006 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 2.0 M 2024-03-25T16:23:10.006 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 26 M 2024-03-25T16:23:10.006 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-03-25T16:23:10.197 INFO:teuthology.orchestra.run.smithi098.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 163 kB/s | 18 kB 00:00 2024-03-25T16:23:10.247 INFO:teuthology.orchestra.run.smithi098.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 151 kB/s | 24 kB 00:00 2024-03-25T16:23:10.298 INFO:teuthology.orchestra.run.smithi098.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.2 MB/s | 174 kB 00:00 2024-03-25T16:23:10.323 INFO:teuthology.orchestra.run.smithi098.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-03-25T16:23:10.348 INFO:teuthology.orchestra.run.smithi098.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-25T16:23:10.373 INFO:teuthology.orchestra.run.smithi098.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 868 kB/s | 43 kB 00:00 2024-03-25T16:23:10.424 INFO:teuthology.orchestra.run.smithi098.stdout:(7/13): python3-certifi-2021.10.8-2.el9.noarch. 299 kB/s | 15 kB 00:00 2024-03-25T16:23:10.449 INFO:teuthology.orchestra.run.smithi098.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 320 kB/s | 32 kB 00:00 2024-03-25T16:23:10.499 INFO:teuthology.orchestra.run.smithi098.stdout:(9/13): python3-google-auth-2.28.2-1.el9.noarch 2.6 MB/s | 203 kB 00:00 2024-03-25T16:23:10.533 INFO:teuthology.orchestra.run.smithi098.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 1.7 MB/s | 59 kB 00:00 2024-03-25T16:23:10.576 INFO:teuthology.orchestra.run.smithi098.stdout:(11/13): python3-kubernetes-26.1.0-2.el9.noarch 8.1 MB/s | 1.0 MB 00:00 2024-03-25T16:23:10.600 INFO:teuthology.orchestra.run.smithi098.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.3 MB/s | 90 kB 00:00 2024-03-25T16:23:10.675 INFO:teuthology.orchestra.run.smithi098.stdout:(13/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el 89 kB/s | 52 kB 00:00 2024-03-25T16:23:10.677 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:23:10.677 INFO:teuthology.orchestra.run.smithi098.stdout:Total 3.0 MB/s | 2.0 MB 00:00 2024-03-25T16:23:10.678 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-03-25T16:23:10.691 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-03-25T16:23:10.691 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-03-25T16:23:10.784 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-03-25T16:23:10.787 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-03-25T16:23:11.297 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-03-25T16:23:11.466 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-25T16:23:11.535 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-25T16:23:11.591 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-25T16:23:11.628 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-25T16:23:11.669 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-25T16:23:11.763 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-25T16:23:11.873 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-25T16:23:11.921 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-25T16:23:12.643 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-25T16:23:12.693 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-25T16:23:12.727 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-25T16:23:12.775 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-25T16:23:12.797 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T16:23:13.580 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T16:23:13.580 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-25T16:23:13.580 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-25T16:23:13.580 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-25T16:23:13.580 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-25T16:23:13.580 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-25T16:23:13.580 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-25T16:23:13.580 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-25T16:23:13.581 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-25T16:23:13.581 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-25T16:23:13.581 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-25T16:23:13.581 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-25T16:23:13.581 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-25T16:23:13.939 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-25T16:23:13.939 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T16:23:13.939 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-03-25T16:23:13.940 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:23:13.940 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-25T16:23:13.940 INFO:teuthology.orchestra.run.smithi098.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-25T16:23:13.940 INFO:teuthology.orchestra.run.smithi098.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-25T16:23:13.940 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-25T16:23:13.940 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-25T16:23:13.940 INFO:teuthology.orchestra.run.smithi098.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-25T16:23:13.940 INFO:teuthology.orchestra.run.smithi098.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-25T16:23:13.940 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-25T16:23:13.940 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-25T16:23:13.940 INFO:teuthology.orchestra.run.smithi098.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-25T16:23:13.940 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-25T16:23:13.940 INFO:teuthology.orchestra.run.smithi098.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-25T16:23:13.941 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T16:23:13.941 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-25T16:23:14.201 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph-mgr-cephadm 2024-03-25T16:23:14.761 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:03:09 ago on Mon 25 Mar 2024 04:20:05 PM UTC. 2024-03-25T16:23:14.911 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-25T16:23:14.911 INFO:teuthology.orchestra.run.smithi098.stdout:======================================================================================== 2024-03-25T16:23:14.911 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repository Size 2024-03-25T16:23:14.912 INFO:teuthology.orchestra.run.smithi098.stdout:======================================================================================== 2024-03-25T16:23:14.912 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-03-25T16:23:14.912 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-25T16:23:14.912 INFO:teuthology.orchestra.run.smithi098.stdout:Installing dependencies: 2024-03-25T16:23:14.912 INFO:teuthology.orchestra.run.smithi098.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-25T16:23:14.912 INFO:teuthology.orchestra.run.smithi098.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-25T16:23:14.912 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-25T16:23:14.912 INFO:teuthology.orchestra.run.smithi098.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-25T16:23:14.912 INFO:teuthology.orchestra.run.smithi098.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-25T16:23:14.912 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T16:23:14.913 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-03-25T16:23:14.913 INFO:teuthology.orchestra.run.smithi098.stdout:======================================================================================== 2024-03-25T16:23:14.913 INFO:teuthology.orchestra.run.smithi098.stdout:Install 6 Packages 2024-03-25T16:23:14.913 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T16:23:14.913 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 6.8 M 2024-03-25T16:23:14.913 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 33 M 2024-03-25T16:23:14.914 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-03-25T16:23:15.136 INFO:teuthology.orchestra.run.smithi098.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 2.3 MB/s | 248 kB 00:00 2024-03-25T16:23:15.303 INFO:teuthology.orchestra.run.smithi098.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 3.7 MB/s | 548 kB 00:00 2024-03-25T16:23:15.337 INFO:teuthology.orchestra.run.smithi098.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 1.7 MB/s | 58 kB 00:00 2024-03-25T16:23:15.370 INFO:teuthology.orchestra.run.smithi098.stdout:(4/6): python3-typing-extensions-4.4.0-2.el9.no 1.5 MB/s | 51 kB 00:00 2024-03-25T16:23:15.579 INFO:teuthology.orchestra.run.smithi098.stdout:(5/6): python3-babel-2.9.1-2.el9.noarch.rpm 11 MB/s | 5.8 MB 00:00 2024-03-25T16:23:15.796 INFO:teuthology.orchestra.run.smithi098.stdout:(6/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 180 kB/s | 138 kB 00:00 2024-03-25T16:23:15.797 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:23:15.797 INFO:teuthology.orchestra.run.smithi098.stdout:Total 7.7 MB/s | 6.8 MB 00:00 2024-03-25T16:23:15.810 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-03-25T16:23:15.820 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-03-25T16:23:15.820 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-03-25T16:23:15.909 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-03-25T16:23:15.910 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-03-25T16:23:16.199 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-03-25T16:23:16.366 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-25T16:23:16.414 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-25T16:23:16.918 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-25T16:23:16.990 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-25T16:23:17.061 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-25T16:23:17.085 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T16:23:17.994 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T16:23:17.994 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-25T16:23:17.995 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-25T16:23:17.995 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-25T16:23:17.995 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-25T16:23:17.995 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-25T16:23:18.296 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-25T16:23:18.296 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T16:23:18.296 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-03-25T16:23:18.296 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:23:18.296 INFO:teuthology.orchestra.run.smithi098.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-25T16:23:18.296 INFO:teuthology.orchestra.run.smithi098.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-25T16:23:18.296 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-25T16:23:18.296 INFO:teuthology.orchestra.run.smithi098.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-25T16:23:18.296 INFO:teuthology.orchestra.run.smithi098.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-25T16:23:18.296 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T16:23:18.296 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-25T16:23:18.534 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph-fuse 2024-03-25T16:23:19.098 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:03:14 ago on Mon 25 Mar 2024 04:20:05 PM UTC. 2024-03-25T16:23:19.247 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-25T16:23:19.248 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T16:23:19.248 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repository Size 2024-03-25T16:23:19.248 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T16:23:19.248 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-03-25T16:23:19.248 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-25T16:23:19.248 INFO:teuthology.orchestra.run.smithi098.stdout:Installing dependencies: 2024-03-25T16:23:19.248 INFO:teuthology.orchestra.run.smithi098.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-25T16:23:19.248 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T16:23:19.248 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-03-25T16:23:19.249 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T16:23:19.249 INFO:teuthology.orchestra.run.smithi098.stdout:Install 2 Packages 2024-03-25T16:23:19.249 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T16:23:19.249 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 964 k 2024-03-25T16:23:19.249 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 2.8 M 2024-03-25T16:23:19.249 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-03-25T16:23:19.366 INFO:teuthology.orchestra.run.smithi098.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 680 kB/s | 79 kB 00:00 2024-03-25T16:23:21.356 INFO:teuthology.orchestra.run.smithi098.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 420 kB/s | 885 kB 00:02 2024-03-25T16:23:21.358 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:23:21.358 INFO:teuthology.orchestra.run.smithi098.stdout:Total 457 kB/s | 964 kB 00:02 2024-03-25T16:23:21.359 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-03-25T16:23:21.381 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-03-25T16:23:21.382 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-03-25T16:23:21.457 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-03-25T16:23:21.457 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-03-25T16:23:21.638 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-03-25T16:23:21.774 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-25T16:23:21.847 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:23:22.496 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:23:22.496 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T16:23:22.824 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-25T16:23:22.824 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T16:23:22.824 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-03-25T16:23:22.824 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-25T16:23:22.824 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T16:23:22.824 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-25T16:23:23.128 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph-volume 2024-03-25T16:23:23.704 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:03:18 ago on Mon 25 Mar 2024 04:20:05 PM UTC. 2024-03-25T16:23:23.852 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-25T16:23:23.853 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T16:23:23.853 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repository Size 2024-03-25T16:23:23.853 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T16:23:23.853 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-03-25T16:23:23.853 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-25T16:23:23.853 INFO:teuthology.orchestra.run.smithi098.stdout:Installing dependencies: 2024-03-25T16:23:23.853 INFO:teuthology.orchestra.run.smithi098.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-25T16:23:23.853 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T16:23:23.853 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-03-25T16:23:23.853 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T16:23:23.853 INFO:teuthology.orchestra.run.smithi098.stdout:Install 2 Packages 2024-03-25T16:23:23.854 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T16:23:23.854 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 332 k 2024-03-25T16:23:23.854 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 1.5 M 2024-03-25T16:23:23.854 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-03-25T16:23:23.987 INFO:teuthology.orchestra.run.smithi098.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 520 kB/s | 69 kB 00:00 2024-03-25T16:23:24.587 INFO:teuthology.orchestra.run.smithi098.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 360 kB/s | 264 kB 00:00 2024-03-25T16:23:24.588 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:23:24.588 INFO:teuthology.orchestra.run.smithi098.stdout:Total 452 kB/s | 332 kB 00:00 2024-03-25T16:23:24.589 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-03-25T16:23:24.597 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-03-25T16:23:24.597 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-03-25T16:23:24.641 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-03-25T16:23:24.641 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-03-25T16:23:24.792 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-03-25T16:23:24.971 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-25T16:23:24.998 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T16:23:25.014 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T16:23:25.014 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:23:25.014 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-25T16:23:25.014 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T16:23:25.742 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-25T16:23:26.089 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-25T16:23:26.089 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T16:23:26.089 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-03-25T16:23:26.089 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:23:26.089 INFO:teuthology.orchestra.run.smithi098.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-25T16:23:26.089 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T16:23:26.089 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-25T16:23:26.419 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install librados-devel 2024-03-25T16:23:26.997 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:03:21 ago on Mon 25 Mar 2024 04:20:05 PM UTC. 2024-03-25T16:23:27.145 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-25T16:23:27.146 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T16:23:27.146 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repo Size 2024-03-25T16:23:27.146 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T16:23:27.146 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-03-25T16:23:27.146 INFO:teuthology.orchestra.run.smithi098.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-25T16:23:27.146 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T16:23:27.146 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-03-25T16:23:27.146 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T16:23:27.146 INFO:teuthology.orchestra.run.smithi098.stdout:Install 1 Package 2024-03-25T16:23:27.146 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T16:23:27.147 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 127 k 2024-03-25T16:23:27.147 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 456 k 2024-03-25T16:23:27.147 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-03-25T16:23:28.519 INFO:teuthology.orchestra.run.smithi098.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 92 kB/s | 127 kB 00:01 2024-03-25T16:23:28.519 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:23:28.520 INFO:teuthology.orchestra.run.smithi098.stdout:Total 92 kB/s | 127 kB 00:01 2024-03-25T16:23:28.520 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-03-25T16:23:28.527 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-03-25T16:23:28.527 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-03-25T16:23:28.589 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-03-25T16:23:28.589 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-03-25T16:23:28.740 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-03-25T16:23:28.851 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:23:29.313 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:23:29.595 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:23:29.595 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T16:23:29.595 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-03-25T16:23:29.595 INFO:teuthology.orchestra.run.smithi098.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:23:29.595 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T16:23:29.596 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-25T16:23:29.847 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install libcephfs2 2024-03-25T16:23:30.424 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:03:25 ago on Mon 25 Mar 2024 04:20:05 PM UTC. 2024-03-25T16:23:30.522 INFO:teuthology.orchestra.run.smithi098.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:23:30.578 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-25T16:23:30.579 INFO:teuthology.orchestra.run.smithi098.stdout:Nothing to do. 2024-03-25T16:23:30.579 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-25T16:23:30.643 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install libcephfs-devel 2024-03-25T16:23:31.208 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:03:26 ago on Mon 25 Mar 2024 04:20:05 PM UTC. 2024-03-25T16:23:31.356 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-25T16:23:31.357 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T16:23:31.357 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repo Size 2024-03-25T16:23:31.357 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T16:23:31.357 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-03-25T16:23:31.357 INFO:teuthology.orchestra.run.smithi098.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-25T16:23:31.357 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T16:23:31.358 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-03-25T16:23:31.358 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T16:23:31.358 INFO:teuthology.orchestra.run.smithi098.stdout:Install 1 Package 2024-03-25T16:23:31.358 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T16:23:31.358 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 31 k 2024-03-25T16:23:31.358 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 140 k 2024-03-25T16:23:31.358 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-03-25T16:23:31.857 INFO:teuthology.orchestra.run.smithi098.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 62 kB/s | 31 kB 00:00 2024-03-25T16:23:31.857 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:23:31.858 INFO:teuthology.orchestra.run.smithi098.stdout:Total 62 kB/s | 31 kB 00:00 2024-03-25T16:23:31.858 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-03-25T16:23:31.861 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-03-25T16:23:31.861 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-03-25T16:23:31.892 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-03-25T16:23:31.892 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-03-25T16:23:32.019 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-03-25T16:23:32.122 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:23:32.484 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:23:32.751 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:23:32.751 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T16:23:32.752 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-03-25T16:23:32.752 INFO:teuthology.orchestra.run.smithi098.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:23:32.752 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T16:23:32.752 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-25T16:23:32.925 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install librados2 2024-03-25T16:23:33.493 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:03:28 ago on Mon 25 Mar 2024 04:20:05 PM UTC. 2024-03-25T16:23:33.590 INFO:teuthology.orchestra.run.smithi098.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:23:33.642 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-25T16:23:33.643 INFO:teuthology.orchestra.run.smithi098.stdout:Nothing to do. 2024-03-25T16:23:33.643 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-25T16:23:33.698 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install librbd1 2024-03-25T16:23:34.267 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:03:29 ago on Mon 25 Mar 2024 04:20:05 PM UTC. 2024-03-25T16:23:34.363 INFO:teuthology.orchestra.run.smithi098.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:23:34.414 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-25T16:23:34.415 INFO:teuthology.orchestra.run.smithi098.stdout:Nothing to do. 2024-03-25T16:23:34.415 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-25T16:23:34.468 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install python3-rados 2024-03-25T16:23:35.031 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:03:30 ago on Mon 25 Mar 2024 04:20:05 PM UTC. 2024-03-25T16:23:35.128 INFO:teuthology.orchestra.run.smithi098.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:23:35.179 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-25T16:23:35.180 INFO:teuthology.orchestra.run.smithi098.stdout:Nothing to do. 2024-03-25T16:23:35.180 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-25T16:23:35.234 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install python3-rgw 2024-03-25T16:23:35.802 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:03:30 ago on Mon 25 Mar 2024 04:20:05 PM UTC. 2024-03-25T16:23:35.898 INFO:teuthology.orchestra.run.smithi098.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:23:35.950 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-25T16:23:35.951 INFO:teuthology.orchestra.run.smithi098.stdout:Nothing to do. 2024-03-25T16:23:35.951 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-25T16:23:36.004 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install python3-cephfs 2024-03-25T16:23:36.563 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:03:31 ago on Mon 25 Mar 2024 04:20:05 PM UTC. 2024-03-25T16:23:36.658 INFO:teuthology.orchestra.run.smithi098.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:23:36.709 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-25T16:23:36.710 INFO:teuthology.orchestra.run.smithi098.stdout:Nothing to do. 2024-03-25T16:23:36.710 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-25T16:23:36.759 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install python3-rbd 2024-03-25T16:23:37.318 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:03:32 ago on Mon 25 Mar 2024 04:20:05 PM UTC. 2024-03-25T16:23:37.414 INFO:teuthology.orchestra.run.smithi098.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:23:37.466 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-25T16:23:37.467 INFO:teuthology.orchestra.run.smithi098.stdout:Nothing to do. 2024-03-25T16:23:37.467 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-25T16:23:37.521 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install rbd-fuse 2024-03-25T16:23:38.086 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:03:33 ago on Mon 25 Mar 2024 04:20:05 PM UTC. 2024-03-25T16:23:38.235 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-25T16:23:38.235 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T16:23:38.235 INFO:teuthology.orchestra.run.smithi098.stdout: Package Architecture Version Repository Size 2024-03-25T16:23:38.235 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T16:23:38.236 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-03-25T16:23:38.236 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-25T16:23:38.236 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T16:23:38.236 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-03-25T16:23:38.236 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T16:23:38.236 INFO:teuthology.orchestra.run.smithi098.stdout:Install 1 Package 2024-03-25T16:23:38.236 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T16:23:38.236 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 85 k 2024-03-25T16:23:38.236 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 231 k 2024-03-25T16:23:38.236 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-03-25T16:23:38.893 INFO:teuthology.orchestra.run.smithi098.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 130 kB/s | 85 kB 00:00 2024-03-25T16:23:38.894 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:23:38.894 INFO:teuthology.orchestra.run.smithi098.stdout:Total 130 kB/s | 85 kB 00:00 2024-03-25T16:23:38.894 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-03-25T16:23:38.901 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-03-25T16:23:38.901 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-03-25T16:23:38.957 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-03-25T16:23:38.957 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-03-25T16:23:39.094 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-03-25T16:23:39.199 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:23:39.584 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:23:39.885 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:23:39.885 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T16:23:39.885 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-03-25T16:23:39.885 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:23:39.885 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T16:23:39.885 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-25T16:23:40.128 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install rbd-mirror 2024-03-25T16:23:40.743 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:03:35 ago on Mon 25 Mar 2024 04:20:05 PM UTC. 2024-03-25T16:23:40.910 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-25T16:23:40.910 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T16:23:40.911 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repo Size 2024-03-25T16:23:40.911 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T16:23:40.911 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-03-25T16:23:40.911 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T16:23:40.911 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T16:23:40.911 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-03-25T16:23:40.911 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T16:23:40.911 INFO:teuthology.orchestra.run.smithi098.stdout:Install 1 Package 2024-03-25T16:23:40.911 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T16:23:40.911 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 3.1 M 2024-03-25T16:23:40.911 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 13 M 2024-03-25T16:23:40.911 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-03-25T16:23:44.380 INFO:teuthology.orchestra.run.smithi098.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 912 kB/s | 3.1 MB 00:03 2024-03-25T16:23:44.381 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:23:44.382 INFO:teuthology.orchestra.run.smithi098.stdout:Total 911 kB/s | 3.1 MB 00:03 2024-03-25T16:23:44.382 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-03-25T16:23:44.393 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-03-25T16:23:44.393 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-03-25T16:23:44.469 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-03-25T16:23:44.469 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-03-25T16:23:44.898 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-03-25T16:23:44.959 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:23:44.981 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:23:44.981 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:23:44.981 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-25T16:23:44.981 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T16:23:44.982 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T16:23:44.982 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T16:23:46.102 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:23:46.102 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T16:23:46.102 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-03-25T16:23:46.102 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:23:46.102 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T16:23:46.102 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-25T16:23:46.333 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install rbd-nbd 2024-03-25T16:23:46.914 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:03:41 ago on Mon 25 Mar 2024 04:20:05 PM UTC. 2024-03-25T16:23:47.064 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-25T16:23:47.065 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T16:23:47.065 INFO:teuthology.orchestra.run.smithi098.stdout: Package Architecture Version Repository Size 2024-03-25T16:23:47.065 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T16:23:47.065 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-03-25T16:23:47.065 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-25T16:23:47.065 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T16:23:47.065 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-03-25T16:23:47.065 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T16:23:47.066 INFO:teuthology.orchestra.run.smithi098.stdout:Install 1 Package 2024-03-25T16:23:47.066 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T16:23:47.066 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 171 k 2024-03-25T16:23:47.066 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 494 k 2024-03-25T16:23:47.066 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-03-25T16:23:48.043 INFO:teuthology.orchestra.run.smithi098.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 175 kB/s | 171 kB 00:00 2024-03-25T16:23:48.044 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:23:48.044 INFO:teuthology.orchestra.run.smithi098.stdout:Total 175 kB/s | 171 kB 00:00 2024-03-25T16:23:48.044 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-03-25T16:23:48.053 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-03-25T16:23:48.053 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-03-25T16:23:48.112 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-03-25T16:23:48.112 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-03-25T16:23:48.261 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-03-25T16:23:48.367 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:23:48.762 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:23:49.049 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:23:49.049 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T16:23:49.049 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-03-25T16:23:49.049 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:23:49.050 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T16:23:49.050 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-25T16:23:49.263 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph-base-debuginfo 2024-03-25T16:23:49.825 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:03:44 ago on Mon 25 Mar 2024 04:20:05 PM UTC. 2024-03-25T16:23:49.977 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-25T16:23:49.977 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T16:23:49.977 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repo Size 2024-03-25T16:23:49.978 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T16:23:49.978 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-03-25T16:23:49.978 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-base-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 68 M 2024-03-25T16:23:49.978 INFO:teuthology.orchestra.run.smithi098.stdout:Installing dependencies: 2024-03-25T16:23:49.978 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 22 M 2024-03-25T16:23:49.978 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T16:23:49.978 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-03-25T16:23:49.978 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T16:23:49.978 INFO:teuthology.orchestra.run.smithi098.stdout:Install 2 Packages 2024-03-25T16:23:49.978 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T16:23:49.979 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 91 M 2024-03-25T16:23:49.979 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 498 M 2024-03-25T16:23:49.979 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-03-25T16:24:23.240 INFO:teuthology.orchestra.run.smithi097.stdout:Extra Packages for Enterprise Linux 79 kB/s | 21 MB 04:26 2024-03-25T16:24:28.157 INFO:teuthology.orchestra.run.smithi097.stdout:lab-extras 12 kB/s | 1.7 kB 00:00 2024-03-25T16:24:30.578 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-03-25T16:24:30.596 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-03-25T16:24:30.596 INFO:teuthology.orchestra.run.smithi097.stdout: Package Arch Version Repository Size 2024-03-25T16:24:30.596 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-03-25T16:24:30.596 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2024-03-25T16:24:30.596 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-25T16:24:30.597 INFO:teuthology.orchestra.run.smithi097.stdout:Upgrading: 2024-03-25T16:24:30.597 INFO:teuthology.orchestra.run.smithi097.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-25T16:24:30.597 INFO:teuthology.orchestra.run.smithi097.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T16:24:30.597 INFO:teuthology.orchestra.run.smithi097.stdout:Installing dependencies: 2024-03-25T16:24:30.597 INFO:teuthology.orchestra.run.smithi097.stdout: boost-program-options 2024-03-25T16:24:30.597 INFO:teuthology.orchestra.run.smithi097.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-25T16:24:30.597 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-25T16:24:30.597 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T16:24:30.597 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-25T16:24:30.597 INFO:teuthology.orchestra.run.smithi097.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-25T16:24:30.597 INFO:teuthology.orchestra.run.smithi097.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-25T16:24:30.597 INFO:teuthology.orchestra.run.smithi097.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-25T16:24:30.598 INFO:teuthology.orchestra.run.smithi097.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-25T16:24:30.598 INFO:teuthology.orchestra.run.smithi097.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-25T16:24:30.598 INFO:teuthology.orchestra.run.smithi097.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-25T16:24:30.598 INFO:teuthology.orchestra.run.smithi097.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-25T16:24:30.598 INFO:teuthology.orchestra.run.smithi097.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-25T16:24:30.598 INFO:teuthology.orchestra.run.smithi097.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-25T16:24:30.598 INFO:teuthology.orchestra.run.smithi097.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-25T16:24:30.598 INFO:teuthology.orchestra.run.smithi097.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-25T16:24:30.598 INFO:teuthology.orchestra.run.smithi097.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-25T16:24:30.598 INFO:teuthology.orchestra.run.smithi097.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-25T16:24:30.598 INFO:teuthology.orchestra.run.smithi097.stdout: python3-ceph-argparse 2024-03-25T16:24:30.598 INFO:teuthology.orchestra.run.smithi097.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-25T16:24:30.599 INFO:teuthology.orchestra.run.smithi097.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-25T16:24:30.599 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-25T16:24:30.599 INFO:teuthology.orchestra.run.smithi097.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-25T16:24:30.599 INFO:teuthology.orchestra.run.smithi097.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-25T16:24:30.599 INFO:teuthology.orchestra.run.smithi097.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-25T16:24:30.599 INFO:teuthology.orchestra.run.smithi097.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-25T16:24:30.599 INFO:teuthology.orchestra.run.smithi097.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-25T16:24:30.599 INFO:teuthology.orchestra.run.smithi097.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-25T16:24:30.599 INFO:teuthology.orchestra.run.smithi097.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-25T16:24:30.599 INFO:teuthology.orchestra.run.smithi097.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-25T16:24:30.599 INFO:teuthology.orchestra.run.smithi097.stdout:Installing weak dependencies: 2024-03-25T16:24:30.599 INFO:teuthology.orchestra.run.smithi097.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-25T16:24:30.599 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T16:24:30.600 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2024-03-25T16:24:30.600 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-03-25T16:24:30.600 INFO:teuthology.orchestra.run.smithi097.stdout:Install 30 Packages 2024-03-25T16:24:30.600 INFO:teuthology.orchestra.run.smithi097.stdout:Upgrade 2 Packages 2024-03-25T16:24:30.600 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T16:24:30.600 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 58 M 2024-03-25T16:24:30.600 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2024-03-25T16:24:41.650 INFO:teuthology.orchestra.run.smithi098.stdout:(1/2): ceph-debuginfo-19.0.0-1439.g5dcb190d.el9 445 kB/s | 22 MB 00:51 2024-03-25T16:24:47.723 INFO:teuthology.orchestra.run.smithi097.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 330 kB/s | 5.3 MB 00:16 2024-03-25T16:24:47.823 INFO:teuthology.orchestra.run.smithi097.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 252 kB/s | 25 kB 00:00 2024-03-25T16:24:49.790 INFO:teuthology.orchestra.run.smithi097.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 377 kB/s | 742 kB 00:01 2024-03-25T16:24:49.940 INFO:teuthology.orchestra.run.smithi097.stdout:(4/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9. 531 kB/s | 9.7 MB 00:18 2024-03-25T16:24:50.999 INFO:teuthology.orchestra.run.smithi097.stdout:(5/32): libradosstriper1-19.0.0-1439.g5dcb190d. 412 kB/s | 498 kB 00:01 2024-03-25T16:24:51.174 INFO:teuthology.orchestra.run.smithi097.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 258 kB/s | 45 kB 00:00 2024-03-25T16:24:51.649 INFO:teuthology.orchestra.run.smithi097.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 276 kB/s | 131 kB 00:00 2024-03-25T16:24:52.141 INFO:teuthology.orchestra.run.smithi097.stdout:(8/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 329 kB/s | 162 kB 00:00 2024-03-25T16:24:53.491 INFO:teuthology.orchestra.run.smithi097.stdout:(9/32): python3-rados-19.0.0-1439.g5dcb190d.el9 237 kB/s | 320 kB 00:01 2024-03-25T16:24:54.283 INFO:teuthology.orchestra.run.smithi097.stdout:(10/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 376 kB/s | 298 kB 00:00 2024-03-25T16:24:54.516 INFO:teuthology.orchestra.run.smithi097.stdout:(11/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 426 kB/s | 99 kB 00:00 2024-03-25T16:24:54.741 INFO:teuthology.orchestra.run.smithi097.stdout:(12/32): boost-program-options-1.75.0-8.el9.x86 470 kB/s | 106 kB 00:00 2024-03-25T16:24:54.858 INFO:teuthology.orchestra.run.smithi097.stdout:(13/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.3 MB/s | 159 kB 00:00 2024-03-25T16:24:54.917 INFO:teuthology.orchestra.run.smithi097.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 759 kB/s | 44 kB 00:00 2024-03-25T16:24:55.042 INFO:teuthology.orchestra.run.smithi097.stdout:(15/32): librdkafka-1.6.1-102.el9.x86_64.rpm 5.2 MB/s | 661 kB 00:00 2024-03-25T16:24:55.101 INFO:teuthology.orchestra.run.smithi097.stdout:(16/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.7 MB/s | 282 kB 00:00 2024-03-25T16:24:55.151 INFO:teuthology.orchestra.run.smithi097.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 825 kB/s | 41 kB 00:00 2024-03-25T16:24:55.385 INFO:teuthology.orchestra.run.smithi097.stdout:(18/32): qatlib-23.11.0-1.el9.x86_64.rpm 928 kB/s | 216 kB 00:00 2024-03-25T16:24:55.444 INFO:teuthology.orchestra.run.smithi097.stdout:(19/32): qatlib-service-23.11.0-1.el9.x86_64.rp 600 kB/s | 35 kB 00:00 2024-03-25T16:24:55.494 INFO:teuthology.orchestra.run.smithi097.stdout:(20/32): qatzip-libs-1.1.2-2.el9.x86_64.rpm 890 kB/s | 44 kB 00:00 2024-03-25T16:24:55.869 INFO:teuthology.orchestra.run.smithi097.stdout:(21/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 824 kB/s | 309 kB 00:00 2024-03-25T16:24:56.144 INFO:teuthology.orchestra.run.smithi097.stdout:(22/32): libarrow-9.0.0-10.el9.x86_64.rpm 16 MB/s | 4.4 MB 00:00 2024-03-25T16:24:56.211 INFO:teuthology.orchestra.run.smithi097.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 391 kB/s | 26 kB 00:00 2024-03-25T16:24:56.278 INFO:teuthology.orchestra.run.smithi097.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 730 kB/s | 49 kB 00:00 2024-03-25T16:24:56.345 INFO:teuthology.orchestra.run.smithi097.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.0 MB/s | 67 kB 00:00 2024-03-25T16:24:56.429 INFO:teuthology.orchestra.run.smithi097.stdout:(26/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 9.8 MB/s | 838 kB 00:00 2024-03-25T16:24:56.496 INFO:teuthology.orchestra.run.smithi097.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 2.8 MB/s | 192 kB 00:00 2024-03-25T16:24:56.588 INFO:teuthology.orchestra.run.smithi097.stdout:(28/32): thrift-0.15.0-2.el9.x86_64.rpm 17 MB/s | 1.6 MB 00:00 2024-03-25T16:24:58.538 INFO:teuthology.orchestra.run.smithi097.stdout:(29/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 744 kB/s | 20 MB 00:27 2024-03-25T16:25:02.604 INFO:teuthology.orchestra.run.smithi097.stdout:(30/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_ 412 kB/s | 5.1 MB 00:12 2024-03-25T16:25:02.654 INFO:teuthology.orchestra.run.smithi097.stdout:(31/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 778 kB/s | 3.1 MB 00:04 2024-03-25T16:25:05.238 INFO:teuthology.orchestra.run.smithi097.stdout:(32/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 399 kB/s | 3.4 MB 00:08 2024-03-25T16:25:05.242 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:25:05.242 INFO:teuthology.orchestra.run.smithi097.stdout:Total 1.7 MB/s | 58 MB 00:34 2024-03-25T16:25:05.274 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2024-03-25T16:25:05.312 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2024-03-25T16:25:05.312 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2024-03-25T16:25:05.790 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2024-03-25T16:25:05.791 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2024-03-25T16:25:07.253 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2024-03-25T16:25:07.341 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-25T16:25:07.375 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-25T16:25:07.401 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-25T16:25:07.423 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-25T16:25:07.440 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-25T16:25:07.649 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-25T16:25:07.705 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-25T16:25:08.022 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-25T16:25:08.086 INFO:teuthology.orchestra.run.smithi097.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:25:08.169 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:25:08.256 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-25T16:25:08.287 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T16:25:08.333 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T16:25:08.361 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-25T16:25:08.405 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-25T16:25:08.473 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T16:25:08.498 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T16:25:08.560 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T16:25:08.587 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 14/34 2024-03-25T16:25:08.697 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-25T16:25:08.731 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-25T16:25:08.778 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-25T16:25:08.813 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-25T16:25:08.841 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-25T16:25:08.874 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-25T16:25:09.189 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-25T16:25:09.237 INFO:teuthology.orchestra.run.smithi097.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T16:25:09.309 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T16:25:09.341 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-25T16:25:09.383 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-25T16:25:09.834 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-25T16:25:09.855 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T16:25:09.926 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T16:25:09.994 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-25T16:25:10.025 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-25T16:25:12.176 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:25:12.221 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:25:12.765 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:25:12.822 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T16:25:12.915 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T16:25:12.915 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-25T16:25:12.915 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-25T16:25:12.915 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T16:25:12.967 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:25:22.217 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:25:22.217 INFO:teuthology.orchestra.run.smithi097.stdout:skipping the directory /sys 2024-03-25T16:25:22.217 INFO:teuthology.orchestra.run.smithi097.stdout:skipping the directory /proc 2024-03-25T16:25:22.217 INFO:teuthology.orchestra.run.smithi097.stdout:skipping the directory /mnt 2024-03-25T16:25:22.217 INFO:teuthology.orchestra.run.smithi097.stdout:skipping the directory /var/tmp 2024-03-25T16:25:22.217 INFO:teuthology.orchestra.run.smithi097.stdout:skipping the directory /home 2024-03-25T16:25:22.217 INFO:teuthology.orchestra.run.smithi097.stdout:skipping the directory /root 2024-03-25T16:25:22.217 INFO:teuthology.orchestra.run.smithi097.stdout:skipping the directory /tmp 2024-03-25T16:25:22.217 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T16:25:23.085 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T16:25:23.109 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T16:25:23.109 INFO:teuthology.orchestra.run.smithi097.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:25:23.109 INFO:teuthology.orchestra.run.smithi097.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-25T16:25:23.109 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T16:25:23.109 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T16:25:23.109 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T16:25:23.111 INFO:teuthology.orchestra.run.smithi097.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T16:25:23.196 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T16:25:23.196 INFO:teuthology.orchestra.run.smithi097.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:25:24.288 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:25:24.288 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-25T16:25:24.288 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-25T16:25:24.288 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-25T16:25:24.288 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-25T16:25:24.288 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-25T16:25:24.288 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-25T16:25:24.288 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:25:24.289 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-25T16:25:24.289 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-25T16:25:24.289 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-25T16:25:24.289 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-25T16:25:24.289 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T16:25:24.289 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-25T16:25:24.289 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-25T16:25:24.289 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-25T16:25:24.289 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-25T16:25:24.289 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-25T16:25:24.289 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-25T16:25:24.289 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-25T16:25:24.289 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 20/34 2024-03-25T16:25:24.289 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 21/34 2024-03-25T16:25:24.290 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 22/34 2024-03-25T16:25:24.290 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-25T16:25:24.290 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-25T16:25:24.290 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-25T16:25:24.290 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-25T16:25:24.290 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-25T16:25:24.291 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-25T16:25:24.291 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-25T16:25:24.291 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-25T16:25:24.291 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:25:24.292 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-25T16:25:24.292 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-25T16:25:25.341 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:25:25.342 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T16:25:25.342 INFO:teuthology.orchestra.run.smithi097.stdout:Upgraded: 2024-03-25T16:25:25.342 INFO:teuthology.orchestra.run.smithi097.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:25:25.342 INFO:teuthology.orchestra.run.smithi097.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:25:25.342 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2024-03-25T16:25:25.342 INFO:teuthology.orchestra.run.smithi097.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-25T16:25:25.342 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:25:25.342 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:25:25.342 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:25:25.342 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:25:25.342 INFO:teuthology.orchestra.run.smithi097.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-25T16:25:25.342 INFO:teuthology.orchestra.run.smithi097.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-25T16:25:25.343 INFO:teuthology.orchestra.run.smithi097.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-25T16:25:25.343 INFO:teuthology.orchestra.run.smithi097.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:25:25.343 INFO:teuthology.orchestra.run.smithi097.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-25T16:25:25.343 INFO:teuthology.orchestra.run.smithi097.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-25T16:25:25.343 INFO:teuthology.orchestra.run.smithi097.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-25T16:25:25.343 INFO:teuthology.orchestra.run.smithi097.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:25:25.343 INFO:teuthology.orchestra.run.smithi097.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-25T16:25:25.343 INFO:teuthology.orchestra.run.smithi097.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:25:25.343 INFO:teuthology.orchestra.run.smithi097.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-25T16:25:25.343 INFO:teuthology.orchestra.run.smithi097.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-25T16:25:25.343 INFO:teuthology.orchestra.run.smithi097.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-25T16:25:25.343 INFO:teuthology.orchestra.run.smithi097.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:25:25.343 INFO:teuthology.orchestra.run.smithi097.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:25:25.343 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:25:25.344 INFO:teuthology.orchestra.run.smithi097.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-25T16:25:25.344 INFO:teuthology.orchestra.run.smithi097.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:25:25.344 INFO:teuthology.orchestra.run.smithi097.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:25:25.344 INFO:teuthology.orchestra.run.smithi097.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:25:25.344 INFO:teuthology.orchestra.run.smithi097.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-25T16:25:25.344 INFO:teuthology.orchestra.run.smithi097.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-25T16:25:25.344 INFO:teuthology.orchestra.run.smithi097.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-25T16:25:25.344 INFO:teuthology.orchestra.run.smithi097.stdout: re2-1:20211101-3.el9.x86_64 2024-03-25T16:25:25.344 INFO:teuthology.orchestra.run.smithi097.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-25T16:25:25.344 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T16:25:25.344 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-03-25T16:25:25.602 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install ceph-test 2024-03-25T16:25:26.159 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:00:58 ago on Mon 25 Mar 2024 04:24:28 PM UTC. 2024-03-25T16:25:26.306 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-03-25T16:25:26.307 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-03-25T16:25:26.307 INFO:teuthology.orchestra.run.smithi097.stdout: Package Arch Version Repository Size 2024-03-25T16:25:26.307 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-03-25T16:25:26.307 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2024-03-25T16:25:26.307 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-25T16:25:26.307 INFO:teuthology.orchestra.run.smithi097.stdout:Installing dependencies: 2024-03-25T16:25:26.307 INFO:teuthology.orchestra.run.smithi097.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-25T16:25:26.307 INFO:teuthology.orchestra.run.smithi097.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-25T16:25:26.307 INFO:teuthology.orchestra.run.smithi097.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-25T16:25:26.307 INFO:teuthology.orchestra.run.smithi097.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-25T16:25:26.308 INFO:teuthology.orchestra.run.smithi097.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-25T16:25:26.308 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T16:25:26.308 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2024-03-25T16:25:26.308 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-03-25T16:25:26.308 INFO:teuthology.orchestra.run.smithi097.stdout:Install 6 Packages 2024-03-25T16:25:26.308 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T16:25:26.308 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 48 M 2024-03-25T16:25:26.309 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 204 M 2024-03-25T16:25:26.309 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2024-03-25T16:25:26.747 INFO:teuthology.orchestra.run.smithi097.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.3 MB/s | 217 kB 00:00 2024-03-25T16:25:26.789 INFO:teuthology.orchestra.run.smithi097.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 7.1 MB/s | 300 kB 00:00 2024-03-25T16:25:26.822 INFO:teuthology.orchestra.run.smithi097.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2024-03-25T16:25:26.906 INFO:teuthology.orchestra.run.smithi097.stdout:(4/6): jq-1.6-16.el9.x86_64.rpm 2.2 MB/s | 187 kB 00:00 2024-03-25T16:25:27.548 INFO:teuthology.orchestra.run.smithi097.stdout:(5/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 168 kB/s | 163 kB 00:00 2024-03-25T16:25:54.425 INFO:teuthology.orchestra.run.smithi098.stdout:(2/2): ceph-base-debuginfo-19.0.0-1439.g5dcb190 561 kB/s | 68 MB 02:04 2024-03-25T16:25:54.426 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:25:54.427 INFO:teuthology.orchestra.run.smithi098.stdout:Total 745 kB/s | 91 MB 02:04 2024-03-25T16:25:54.427 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-03-25T16:25:54.431 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-03-25T16:25:54.431 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-03-25T16:25:55.013 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-03-25T16:25:55.013 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-03-25T16:25:58.111 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-03-25T16:26:05.247 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T16:26:05.318 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T16:26:08.760 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T16:26:08.760 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/2 2024-03-25T16:26:09.030 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:26:09.031 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T16:26:09.031 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-03-25T16:26:09.031 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:26:09.031 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:26:09.031 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T16:26:09.031 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-25T16:26:09.251 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph-common-debuginfo 2024-03-25T16:26:09.825 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:06:04 ago on Mon 25 Mar 2024 04:20:05 PM UTC. 2024-03-25T16:26:09.983 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-25T16:26:09.984 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T16:26:09.984 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repo Size 2024-03-25T16:26:09.984 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T16:26:09.984 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-03-25T16:26:09.985 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-common-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 416 M 2024-03-25T16:26:09.985 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T16:26:09.985 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-03-25T16:26:09.985 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T16:26:09.985 INFO:teuthology.orchestra.run.smithi098.stdout:Install 1 Package 2024-03-25T16:26:09.985 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T16:26:09.985 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 416 M 2024-03-25T16:26:09.985 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 1.7 G 2024-03-25T16:26:09.985 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-03-25T16:27:02.663 INFO:teuthology.orchestra.run.smithi097.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 504 kB/s | 47 MB 01:36 2024-03-25T16:27:02.665 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:27:02.665 INFO:teuthology.orchestra.run.smithi097.stdout:Total 512 kB/s | 48 MB 01:36 2024-03-25T16:27:02.676 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2024-03-25T16:27:02.696 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2024-03-25T16:27:02.696 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2024-03-25T16:27:03.075 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2024-03-25T16:27:03.075 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2024-03-25T16:27:03.668 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2024-03-25T16:27:03.729 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-25T16:27:03.773 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-25T16:27:03.813 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T16:27:03.859 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-25T16:27:03.882 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T16:27:08.317 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T16:27:08.390 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T16:27:10.782 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T16:27:10.782 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-25T16:27:10.782 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-25T16:27:10.782 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T16:27:10.783 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-25T16:27:10.783 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-25T16:27:11.154 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-25T16:27:11.155 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T16:27:11.155 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2024-03-25T16:27:11.155 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:27:11.155 INFO:teuthology.orchestra.run.smithi097.stdout: jq-1.6-16.el9.x86_64 2024-03-25T16:27:11.155 INFO:teuthology.orchestra.run.smithi097.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:27:11.155 INFO:teuthology.orchestra.run.smithi097.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-25T16:27:11.155 INFO:teuthology.orchestra.run.smithi097.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-25T16:27:11.155 INFO:teuthology.orchestra.run.smithi097.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-25T16:27:11.155 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T16:27:11.155 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-03-25T16:27:11.511 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install ceph 2024-03-25T16:27:12.080 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:02:44 ago on Mon 25 Mar 2024 04:24:28 PM UTC. 2024-03-25T16:27:12.237 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-03-25T16:27:12.240 INFO:teuthology.orchestra.run.smithi097.stdout:========================================================================================= 2024-03-25T16:27:12.240 INFO:teuthology.orchestra.run.smithi097.stdout: Package Arch Version Repository Size 2024-03-25T16:27:12.240 INFO:teuthology.orchestra.run.smithi097.stdout:========================================================================================= 2024-03-25T16:27:12.240 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2024-03-25T16:27:12.240 INFO:teuthology.orchestra.run.smithi097.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-25T16:27:12.240 INFO:teuthology.orchestra.run.smithi097.stdout:Installing dependencies: 2024-03-25T16:27:12.240 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-25T16:27:12.240 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-25T16:27:12.240 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-25T16:27:12.240 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-25T16:27:12.241 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-25T16:27:12.241 INFO:teuthology.orchestra.run.smithi097.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-25T16:27:12.241 INFO:teuthology.orchestra.run.smithi097.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-25T16:27:12.241 INFO:teuthology.orchestra.run.smithi097.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-25T16:27:12.241 INFO:teuthology.orchestra.run.smithi097.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-25T16:27:12.241 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-25T16:27:12.241 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-25T16:27:12.241 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-25T16:27:12.241 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-25T16:27:12.241 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-25T16:27:12.242 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-25T16:27:12.242 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-25T16:27:12.242 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-25T16:27:12.242 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-25T16:27:12.242 INFO:teuthology.orchestra.run.smithi097.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-25T16:27:12.242 INFO:teuthology.orchestra.run.smithi097.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-25T16:27:12.242 INFO:teuthology.orchestra.run.smithi097.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-25T16:27:12.242 INFO:teuthology.orchestra.run.smithi097.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-25T16:27:12.242 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-25T16:27:12.242 INFO:teuthology.orchestra.run.smithi097.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-25T16:27:12.242 INFO:teuthology.orchestra.run.smithi097.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-25T16:27:12.243 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-25T16:27:12.243 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-25T16:27:12.243 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-25T16:27:12.243 INFO:teuthology.orchestra.run.smithi097.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-25T16:27:12.243 INFO:teuthology.orchestra.run.smithi097.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-25T16:27:12.243 INFO:teuthology.orchestra.run.smithi097.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-25T16:27:12.243 INFO:teuthology.orchestra.run.smithi097.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-25T16:27:12.243 INFO:teuthology.orchestra.run.smithi097.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-25T16:27:12.243 INFO:teuthology.orchestra.run.smithi097.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-25T16:27:12.243 INFO:teuthology.orchestra.run.smithi097.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-25T16:27:12.243 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T16:27:12.244 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2024-03-25T16:27:12.244 INFO:teuthology.orchestra.run.smithi097.stdout:========================================================================================= 2024-03-25T16:27:12.244 INFO:teuthology.orchestra.run.smithi097.stdout:Install 36 Packages 2024-03-25T16:27:12.244 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T16:27:12.244 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 30 M 2024-03-25T16:27:12.244 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 107 M 2024-03-25T16:27:12.245 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2024-03-25T16:27:13.200 INFO:teuthology.orchestra.run.smithi097.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 14 kB/s | 6.6 kB 00:00 2024-03-25T16:27:15.150 INFO:teuthology.orchestra.run.smithi097.stdout:(2/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 631 kB/s | 1.5 MB 00:02 2024-03-25T16:27:16.600 INFO:teuthology.orchestra.run.smithi097.stdout:(3/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 562 kB/s | 2.1 MB 00:03 2024-03-25T16:27:16.984 INFO:teuthology.orchestra.run.smithi097.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 646 kB/s | 247 kB 00:00 2024-03-25T16:27:17.284 INFO:teuthology.orchestra.run.smithi097.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 623 kB/s | 187 kB 00:00 2024-03-25T16:27:17.368 INFO:teuthology.orchestra.run.smithi097.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 1.7 MB/s | 144 kB 00:00 2024-03-25T16:27:17.426 INFO:teuthology.orchestra.run.smithi097.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 542 kB/s | 32 kB 00:00 2024-03-25T16:27:17.477 INFO:teuthology.orchestra.run.smithi097.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 947 kB/s | 47 kB 00:00 2024-03-25T16:27:17.527 INFO:teuthology.orchestra.run.smithi097.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 774 kB/s | 39 kB 00:00 2024-03-25T16:27:17.594 INFO:teuthology.orchestra.run.smithi097.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.5 MB/s | 241 kB 00:00 2024-03-25T16:27:17.669 INFO:teuthology.orchestra.run.smithi097.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 15 MB/s | 1.1 MB 00:00 2024-03-25T16:27:17.703 INFO:teuthology.orchestra.run.smithi097.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 3.0 MB/s | 103 kB 00:00 2024-03-25T16:27:17.736 INFO:teuthology.orchestra.run.smithi097.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 3.7 MB/s | 124 kB 00:00 2024-03-25T16:27:17.820 INFO:teuthology.orchestra.run.smithi097.stdout:(14/36): python3-urllib3-1.26.5-5.el9.noarch.rp 2.2 MB/s | 187 kB 00:00 2024-03-25T16:27:18.012 INFO:teuthology.orchestra.run.smithi097.stdout:(15/36): python3-requests-2.25.1-8.el9.noarch.r 652 kB/s | 125 kB 00:00 2024-03-25T16:27:18.062 INFO:teuthology.orchestra.run.smithi097.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 448 kB/s | 22 kB 00:00 2024-03-25T16:27:18.296 INFO:teuthology.orchestra.run.smithi097.stdout:(17/36): luarocks-3.9.2-1.el9.noarch.rpm 647 kB/s | 151 kB 00:00 2024-03-25T16:27:18.406 INFO:teuthology.orchestra.run.smithi097.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 401 kB/s | 43 kB 00:00 2024-03-25T16:27:18.538 INFO:teuthology.orchestra.run.smithi097.stdout:(19/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.3 MB/s | 172 kB 00:00 2024-03-25T16:27:18.680 INFO:teuthology.orchestra.run.smithi097.stdout:(20/36): python3-cherrypy-18.6.1-2.el9.noarch.r 2.5 MB/s | 358 kB 00:00 2024-03-25T16:27:18.756 INFO:teuthology.orchestra.run.smithi097.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 142 kB/s | 11 kB 00:00 2024-03-25T16:27:18.881 INFO:teuthology.orchestra.run.smithi097.stdout:(22/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86 844 kB/s | 4.7 MB 00:05 2024-03-25T16:27:18.906 INFO:teuthology.orchestra.run.smithi097.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 118 kB/s | 18 kB 00:00 2024-03-25T16:27:18.998 INFO:teuthology.orchestra.run.smithi097.stdout:(24/36): python3-jaraco-functools-3.5.0-2.el9.n 212 kB/s | 19 kB 00:00 2024-03-25T16:27:19.023 INFO:teuthology.orchestra.run.smithi097.stdout:(25/36): python3-jaraco-collections-3.0.0-8.el9 163 kB/s | 23 kB 00:00 2024-03-25T16:27:19.099 INFO:teuthology.orchestra.run.smithi097.stdout:(26/36): python3-jaraco-text-3.2.0-6.el9.noarch 196 kB/s | 20 kB 00:00 2024-03-25T16:27:19.149 INFO:teuthology.orchestra.run.smithi097.stdout:(27/36): python3-logutils-0.3.5-21.el9.noarch.r 370 kB/s | 46 kB 00:00 2024-03-25T16:27:19.199 INFO:teuthology.orchestra.run.smithi097.stdout:(28/36): python3-more-itertools-8.12.0-2.el9.no 788 kB/s | 79 kB 00:00 2024-03-25T16:27:19.324 INFO:teuthology.orchestra.run.smithi097.stdout:(29/36): python3-pecan-1.4.2-3.el9.noarch.rpm 1.5 MB/s | 272 kB 00:00 2024-03-25T16:27:19.350 INFO:teuthology.orchestra.run.smithi097.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 109 kB/s | 16 kB 00:00 2024-03-25T16:27:19.433 INFO:teuthology.orchestra.run.smithi097.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 828 kB/s | 90 kB 00:00 2024-03-25T16:27:19.467 INFO:teuthology.orchestra.run.smithi097.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 305 kB/s | 36 kB 00:00 2024-03-25T16:27:19.534 INFO:teuthology.orchestra.run.smithi097.stdout:(33/36): python3-webob-1.8.7-6.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-03-25T16:27:19.609 INFO:teuthology.orchestra.run.smithi097.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 266 kB/s | 20 kB 00:00 2024-03-25T16:27:19.643 INFO:teuthology.orchestra.run.smithi097.stdout:(35/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 2.4 MB/s | 427 kB 00:00 2024-03-25T16:27:48.565 INFO:teuthology.orchestra.run.smithi097.stdout:(36/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 514 kB/s | 17 MB 00:33 2024-03-25T16:27:48.568 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:27:48.568 INFO:teuthology.orchestra.run.smithi097.stdout:Total 836 kB/s | 30 MB 00:36 2024-03-25T16:27:48.592 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2024-03-25T16:27:48.634 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2024-03-25T16:27:48.634 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2024-03-25T16:27:49.045 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2024-03-25T16:27:49.045 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2024-03-25T16:27:50.349 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2024-03-25T16:27:50.418 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-25T16:27:50.459 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-25T16:27:50.487 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-25T16:27:50.557 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-25T16:27:50.594 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-25T16:27:50.627 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-25T16:27:50.718 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-25T16:27:50.806 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-25T16:27:50.846 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-25T16:27:50.918 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-25T16:27:50.972 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-25T16:27:51.010 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-25T16:27:51.084 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-25T16:27:51.139 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-25T16:27:51.324 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-25T16:27:51.373 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-25T16:27:51.403 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-25T16:27:51.438 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-25T16:27:51.468 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-25T16:27:51.545 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-25T16:27:51.579 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T16:27:51.612 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-25T16:27:51.652 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-25T16:27:51.687 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-25T16:27:51.722 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-25T16:27:51.811 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-25T16:27:51.852 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-25T16:27:51.925 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-25T16:27:52.070 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-25T16:27:52.196 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T16:27:52.353 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-25T16:27:52.383 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T16:27:52.410 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T16:27:52.410 INFO:teuthology.orchestra.run.smithi097.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:27:52.410 INFO:teuthology.orchestra.run.smithi097.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-25T16:27:52.410 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T16:27:52.410 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T16:27:52.410 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T16:27:53.885 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T16:27:53.912 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T16:27:53.912 INFO:teuthology.orchestra.run.smithi097.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:27:53.912 INFO:teuthology.orchestra.run.smithi097.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-25T16:27:53.912 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T16:27:53.913 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T16:27:53.913 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T16:27:54.352 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T16:27:54.376 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T16:27:54.376 INFO:teuthology.orchestra.run.smithi097.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:27:54.376 INFO:teuthology.orchestra.run.smithi097.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-25T16:27:54.376 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T16:27:54.377 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T16:27:54.377 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T16:27:54.608 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T16:27:54.631 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T16:27:54.631 INFO:teuthology.orchestra.run.smithi097.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:27:54.631 INFO:teuthology.orchestra.run.smithi097.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-25T16:27:54.632 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T16:27:54.632 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T16:27:54.632 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T16:27:54.701 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T16:27:56.326 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T16:27:56.326 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-25T16:27:56.326 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-25T16:27:56.326 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-25T16:27:56.326 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-25T16:27:56.326 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-25T16:27:56.326 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-25T16:27:56.327 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-25T16:27:56.327 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-25T16:27:56.327 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-25T16:27:56.327 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-25T16:27:56.327 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-25T16:27:56.327 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-25T16:27:56.327 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-25T16:27:56.327 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-25T16:27:56.327 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-25T16:27:56.327 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-25T16:27:56.327 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-25T16:27:56.327 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-25T16:27:56.328 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-25T16:27:56.328 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-25T16:27:56.328 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T16:27:56.328 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-25T16:27:56.328 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-25T16:27:56.328 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-25T16:27:56.328 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-25T16:27:56.329 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-25T16:27:56.329 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-25T16:27:56.329 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-25T16:27:56.330 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-25T16:27:56.330 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T16:27:56.330 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-25T16:27:56.330 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-25T16:27:56.330 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-25T16:27:56.330 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-25T16:27:56.330 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-25T16:27:57.143 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-25T16:27:57.144 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T16:27:57.144 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2024-03-25T16:27:57.144 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:27:57.144 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:27:57.144 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:27:57.144 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:27:57.144 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:27:57.144 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:27:57.144 INFO:teuthology.orchestra.run.smithi097.stdout: lua-5.4.4-4.el9.x86_64 2024-03-25T16:27:57.144 INFO:teuthology.orchestra.run.smithi097.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-25T16:27:57.144 INFO:teuthology.orchestra.run.smithi097.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-25T16:27:57.145 INFO:teuthology.orchestra.run.smithi097.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-25T16:27:57.145 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-25T16:27:57.145 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-25T16:27:57.145 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-25T16:27:57.145 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-25T16:27:57.145 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-25T16:27:57.145 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-25T16:27:57.145 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-25T16:27:57.145 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-25T16:27:57.145 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-25T16:27:57.145 INFO:teuthology.orchestra.run.smithi097.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-25T16:27:57.145 INFO:teuthology.orchestra.run.smithi097.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-25T16:27:57.145 INFO:teuthology.orchestra.run.smithi097.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-25T16:27:57.145 INFO:teuthology.orchestra.run.smithi097.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-25T16:27:57.146 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-25T16:27:57.146 INFO:teuthology.orchestra.run.smithi097.stdout: python3-ply-3.11-14.el9.noarch 2024-03-25T16:27:57.146 INFO:teuthology.orchestra.run.smithi097.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-25T16:27:57.146 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-25T16:27:57.146 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-25T16:27:57.146 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-25T16:27:57.146 INFO:teuthology.orchestra.run.smithi097.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-25T16:27:57.146 INFO:teuthology.orchestra.run.smithi097.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-25T16:27:57.146 INFO:teuthology.orchestra.run.smithi097.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-25T16:27:57.146 INFO:teuthology.orchestra.run.smithi097.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-25T16:27:57.146 INFO:teuthology.orchestra.run.smithi097.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-25T16:27:57.146 INFO:teuthology.orchestra.run.smithi097.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-25T16:27:57.146 INFO:teuthology.orchestra.run.smithi097.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-25T16:27:57.146 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T16:27:57.147 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-03-25T16:27:57.516 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install ceph-base 2024-03-25T16:27:58.082 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:03:30 ago on Mon 25 Mar 2024 04:24:28 PM UTC. 2024-03-25T16:27:58.182 INFO:teuthology.orchestra.run.smithi097.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:27:58.232 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-03-25T16:27:58.233 INFO:teuthology.orchestra.run.smithi097.stdout:Nothing to do. 2024-03-25T16:27:58.233 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-03-25T16:27:58.307 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install cephadm 2024-03-25T16:27:58.864 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:03:30 ago on Mon 25 Mar 2024 04:24:28 PM UTC. 2024-03-25T16:27:59.012 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-03-25T16:27:59.013 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-03-25T16:27:59.013 INFO:teuthology.orchestra.run.smithi097.stdout: Package Arch Version Repository Size 2024-03-25T16:27:59.013 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-03-25T16:27:59.013 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2024-03-25T16:27:59.013 INFO:teuthology.orchestra.run.smithi097.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-25T16:27:59.014 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T16:27:59.014 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2024-03-25T16:27:59.014 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-03-25T16:27:59.014 INFO:teuthology.orchestra.run.smithi097.stdout:Install 1 Package 2024-03-25T16:27:59.014 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T16:27:59.014 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 747 k 2024-03-25T16:27:59.014 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 754 k 2024-03-25T16:27:59.014 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2024-03-25T16:28:00.988 INFO:teuthology.orchestra.run.smithi097.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 379 kB/s | 747 kB 00:01 2024-03-25T16:28:00.989 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:28:00.990 INFO:teuthology.orchestra.run.smithi097.stdout:Total 378 kB/s | 747 kB 00:01 2024-03-25T16:28:00.990 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2024-03-25T16:28:00.996 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2024-03-25T16:28:00.996 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2024-03-25T16:28:01.002 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2024-03-25T16:28:01.003 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2024-03-25T16:28:01.081 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2024-03-25T16:28:01.576 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:28:01.677 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:28:02.073 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:28:02.308 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:28:02.309 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T16:28:02.309 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2024-03-25T16:28:02.309 INFO:teuthology.orchestra.run.smithi097.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:28:02.309 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T16:28:02.309 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-03-25T16:28:02.527 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install ceph-immutable-object-cache 2024-03-25T16:28:03.094 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:03:35 ago on Mon 25 Mar 2024 04:24:28 PM UTC. 2024-03-25T16:28:03.242 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-03-25T16:28:03.243 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-03-25T16:28:03.243 INFO:teuthology.orchestra.run.smithi097.stdout: Package Arch Version Repo Size 2024-03-25T16:28:03.243 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-03-25T16:28:03.243 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2024-03-25T16:28:03.243 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-25T16:28:03.243 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T16:28:03.243 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2024-03-25T16:28:03.243 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-03-25T16:28:03.243 INFO:teuthology.orchestra.run.smithi097.stdout:Install 1 Package 2024-03-25T16:28:03.243 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T16:28:03.243 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 145 k 2024-03-25T16:28:03.243 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 436 k 2024-03-25T16:28:03.244 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2024-03-25T16:28:03.912 INFO:teuthology.orchestra.run.smithi097.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 218 kB/s | 145 kB 00:00 2024-03-25T16:28:03.912 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:28:03.913 INFO:teuthology.orchestra.run.smithi097.stdout:Total 217 kB/s | 145 kB 00:00 2024-03-25T16:28:03.913 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2024-03-25T16:28:03.920 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2024-03-25T16:28:03.920 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2024-03-25T16:28:03.974 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2024-03-25T16:28:03.974 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2024-03-25T16:28:04.110 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2024-03-25T16:28:04.179 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:28:04.204 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:28:04.204 INFO:teuthology.orchestra.run.smithi097.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:28:04.204 INFO:teuthology.orchestra.run.smithi097.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-25T16:28:04.204 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T16:28:05.185 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:28:05.185 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T16:28:05.185 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2024-03-25T16:28:05.185 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:28:05.185 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T16:28:05.185 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-03-25T16:28:05.460 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install ceph-mgr 2024-03-25T16:28:06.035 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:03:38 ago on Mon 25 Mar 2024 04:24:28 PM UTC. 2024-03-25T16:28:06.134 INFO:teuthology.orchestra.run.smithi097.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:28:06.185 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-03-25T16:28:06.186 INFO:teuthology.orchestra.run.smithi097.stdout:Nothing to do. 2024-03-25T16:28:06.186 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-03-25T16:28:06.257 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install ceph-mgr-dashboard 2024-03-25T16:28:06.813 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:03:38 ago on Mon 25 Mar 2024 04:24:28 PM UTC. 2024-03-25T16:28:06.964 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-03-25T16:28:06.965 INFO:teuthology.orchestra.run.smithi097.stdout:====================================================================================== 2024-03-25T16:28:06.965 INFO:teuthology.orchestra.run.smithi097.stdout: Package Arch Version Repository Size 2024-03-25T16:28:06.965 INFO:teuthology.orchestra.run.smithi097.stdout:====================================================================================== 2024-03-25T16:28:06.965 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2024-03-25T16:28:06.965 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-25T16:28:06.965 INFO:teuthology.orchestra.run.smithi097.stdout:Installing dependencies: 2024-03-25T16:28:06.965 INFO:teuthology.orchestra.run.smithi097.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-25T16:28:06.965 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-25T16:28:06.965 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-25T16:28:06.965 INFO:teuthology.orchestra.run.smithi097.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-25T16:28:06.965 INFO:teuthology.orchestra.run.smithi097.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-25T16:28:06.966 INFO:teuthology.orchestra.run.smithi097.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-25T16:28:06.966 INFO:teuthology.orchestra.run.smithi097.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-25T16:28:06.966 INFO:teuthology.orchestra.run.smithi097.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-25T16:28:06.966 INFO:teuthology.orchestra.run.smithi097.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-25T16:28:06.966 INFO:teuthology.orchestra.run.smithi097.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-25T16:28:06.966 INFO:teuthology.orchestra.run.smithi097.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-25T16:28:06.966 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T16:28:06.966 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2024-03-25T16:28:06.966 INFO:teuthology.orchestra.run.smithi097.stdout:====================================================================================== 2024-03-25T16:28:06.966 INFO:teuthology.orchestra.run.smithi097.stdout:Install 12 Packages 2024-03-25T16:28:06.966 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T16:28:06.967 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 6.9 M 2024-03-25T16:28:06.967 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 26 M 2024-03-25T16:28:06.967 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2024-03-25T16:28:07.658 INFO:teuthology.orchestra.run.smithi097.stdout:(1/12): ceph-grafana-dashboards-19.0.0-1439.g5d 63 kB/s | 31 kB 00:00 2024-03-25T16:28:07.775 INFO:teuthology.orchestra.run.smithi097.stdout:(2/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 25 kB/s | 15 kB 00:00 2024-03-25T16:28:07.825 INFO:teuthology.orchestra.run.smithi097.stdout:(3/12): protobuf-3.14.0-13.el9.x86_64.rpm 6.0 MB/s | 1.0 MB 00:00 2024-03-25T16:28:07.850 INFO:teuthology.orchestra.run.smithi097.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 3.1 MB/s | 238 kB 00:00 2024-03-25T16:28:08.042 INFO:teuthology.orchestra.run.smithi097.stdout:(5/12): protobuf-compiler-3.14.0-13.el9.x86_64. 3.9 MB/s | 863 kB 00:00 2024-03-25T16:28:08.118 INFO:teuthology.orchestra.run.smithi097.stdout:(6/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 2.0 MB/s | 548 kB 00:00 2024-03-25T16:28:08.134 INFO:teuthology.orchestra.run.smithi097.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 212 kB/s | 19 kB 00:00 2024-03-25T16:28:08.227 INFO:teuthology.orchestra.run.smithi097.stdout:(8/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 19 MB/s | 2.0 MB 00:00 2024-03-25T16:28:08.252 INFO:teuthology.orchestra.run.smithi097.stdout:(9/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 1.2 MB/s | 144 kB 00:00 2024-03-25T16:28:08.277 INFO:teuthology.orchestra.run.smithi097.stdout:(10/12): python3-repoze-lru-0.7-16.el9.noarch.r 616 kB/s | 31 kB 00:00 2024-03-25T16:28:08.302 INFO:teuthology.orchestra.run.smithi097.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 3.7 MB/s | 188 kB 00:00 2024-03-25T16:28:09.944 INFO:teuthology.orchestra.run.smithi097.stdout:(12/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb19 655 kB/s | 1.8 MB 00:02 2024-03-25T16:28:09.948 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:28:09.948 INFO:teuthology.orchestra.run.smithi097.stdout:Total 2.3 MB/s | 6.9 MB 00:02 2024-03-25T16:28:09.969 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2024-03-25T16:28:09.988 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2024-03-25T16:28:09.988 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2024-03-25T16:28:10.155 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2024-03-25T16:28:10.155 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2024-03-25T16:28:10.666 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2024-03-25T16:28:10.736 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-25T16:28:10.771 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-25T16:28:10.818 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-25T16:28:10.849 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-25T16:28:10.948 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-25T16:28:11.083 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-25T16:28:11.176 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-25T16:28:11.245 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-25T16:28:11.274 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-25T16:28:11.305 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-25T16:28:11.547 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-25T16:28:11.580 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T16:28:12.399 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T16:28:12.399 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-25T16:28:12.399 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-25T16:28:12.400 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-25T16:28:12.400 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-25T16:28:12.400 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-25T16:28:12.400 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-25T16:28:12.400 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-25T16:28:12.400 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-25T16:28:12.400 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-25T16:28:12.400 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-25T16:28:12.400 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-25T16:28:12.742 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-25T16:28:12.742 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T16:28:12.742 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2024-03-25T16:28:12.742 INFO:teuthology.orchestra.run.smithi097.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-25T16:28:12.742 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:28:12.742 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:28:12.742 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:28:12.742 INFO:teuthology.orchestra.run.smithi097.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-25T16:28:12.742 INFO:teuthology.orchestra.run.smithi097.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-25T16:28:12.742 INFO:teuthology.orchestra.run.smithi097.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-25T16:28:12.742 INFO:teuthology.orchestra.run.smithi097.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-25T16:28:12.743 INFO:teuthology.orchestra.run.smithi097.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-25T16:28:12.743 INFO:teuthology.orchestra.run.smithi097.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-25T16:28:12.743 INFO:teuthology.orchestra.run.smithi097.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-25T16:28:12.743 INFO:teuthology.orchestra.run.smithi097.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-25T16:28:12.743 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T16:28:12.743 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-03-25T16:28:12.979 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-25T16:28:13.533 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:03:45 ago on Mon 25 Mar 2024 04:24:28 PM UTC. 2024-03-25T16:28:13.684 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-03-25T16:28:13.687 INFO:teuthology.orchestra.run.smithi097.stdout:============================================================================================ 2024-03-25T16:28:13.687 INFO:teuthology.orchestra.run.smithi097.stdout: Package Arch Version Repository Size 2024-03-25T16:28:13.687 INFO:teuthology.orchestra.run.smithi097.stdout:============================================================================================ 2024-03-25T16:28:13.687 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2024-03-25T16:28:13.687 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-25T16:28:13.687 INFO:teuthology.orchestra.run.smithi097.stdout:Installing dependencies: 2024-03-25T16:28:13.687 INFO:teuthology.orchestra.run.smithi097.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-25T16:28:13.687 INFO:teuthology.orchestra.run.smithi097.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-25T16:28:13.687 INFO:teuthology.orchestra.run.smithi097.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-25T16:28:13.687 INFO:teuthology.orchestra.run.smithi097.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-25T16:28:13.688 INFO:teuthology.orchestra.run.smithi097.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-25T16:28:13.688 INFO:teuthology.orchestra.run.smithi097.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-25T16:28:13.688 INFO:teuthology.orchestra.run.smithi097.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-25T16:28:13.688 INFO:teuthology.orchestra.run.smithi097.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-25T16:28:13.688 INFO:teuthology.orchestra.run.smithi097.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-25T16:28:13.688 INFO:teuthology.orchestra.run.smithi097.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-25T16:28:13.688 INFO:teuthology.orchestra.run.smithi097.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-25T16:28:13.688 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T16:28:13.688 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2024-03-25T16:28:13.688 INFO:teuthology.orchestra.run.smithi097.stdout:============================================================================================ 2024-03-25T16:28:13.689 INFO:teuthology.orchestra.run.smithi097.stdout:Install 12 Packages 2024-03-25T16:28:13.689 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T16:28:13.689 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 37 M 2024-03-25T16:28:13.689 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 209 M 2024-03-25T16:28:13.689 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2024-03-25T16:28:14.068 INFO:teuthology.orchestra.run.smithi097.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 230 kB/s | 32 kB 00:00 2024-03-25T16:28:14.102 INFO:teuthology.orchestra.run.smithi097.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 520 kB/s | 17 kB 00:00 2024-03-25T16:28:14.153 INFO:teuthology.orchestra.run.smithi097.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 696 kB/s | 34 kB 00:00 2024-03-25T16:28:14.211 INFO:teuthology.orchestra.run.smithi097.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 11 MB/s | 3.0 MB 00:00 2024-03-25T16:28:14.544 INFO:teuthology.orchestra.run.smithi097.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 12 MB/s | 4.7 MB 00:00 2024-03-25T16:28:14.611 INFO:teuthology.orchestra.run.smithi097.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 12 MB/s | 4.8 MB 00:00 2024-03-25T16:28:14.636 INFO:teuthology.orchestra.run.smithi097.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.4 MB/s | 217 kB 00:00 2024-03-25T16:28:15.174 INFO:teuthology.orchestra.run.smithi097.stdout:(8/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.5 MB/s | 803 kB 00:00 2024-03-25T16:28:15.329 INFO:teuthology.orchestra.run.smithi097.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 22 MB/s | 16 MB 00:00 2024-03-25T16:28:15.354 INFO:teuthology.orchestra.run.smithi097.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 1.0 MB/s | 195 kB 00:00 2024-03-25T16:28:15.571 INFO:teuthology.orchestra.run.smithi097.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.0 MB/s | 245 kB 00:00 2024-03-25T16:28:31.261 INFO:teuthology.orchestra.run.smithi097.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 437 kB/s | 7.4 MB 00:17 2024-03-25T16:28:31.263 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:28:31.263 INFO:teuthology.orchestra.run.smithi097.stdout:Total 2.1 MB/s | 37 MB 00:17 2024-03-25T16:28:31.303 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2024-03-25T16:28:31.326 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2024-03-25T16:28:31.326 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2024-03-25T16:28:31.689 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2024-03-25T16:28:31.689 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2024-03-25T16:28:32.384 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2024-03-25T16:28:32.467 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-25T16:28:32.499 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-25T16:28:32.592 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-25T16:28:32.635 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-25T16:28:32.813 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T16:28:32.842 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T16:28:32.927 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-25T16:28:33.568 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-25T16:28:33.688 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-25T16:28:34.924 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-25T16:28:35.965 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-25T16:28:35.993 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T16:28:38.510 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T16:28:38.510 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-25T16:28:38.510 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-25T16:28:38.510 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-25T16:28:38.510 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-25T16:28:38.511 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T16:28:38.511 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T16:28:38.511 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-25T16:28:38.511 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-25T16:28:38.511 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-25T16:28:38.511 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-25T16:28:38.511 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-25T16:28:38.901 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-25T16:28:38.901 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T16:28:38.901 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2024-03-25T16:28:38.901 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:28:38.902 INFO:teuthology.orchestra.run.smithi097.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-25T16:28:38.902 INFO:teuthology.orchestra.run.smithi097.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-25T16:28:38.902 INFO:teuthology.orchestra.run.smithi097.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-25T16:28:38.902 INFO:teuthology.orchestra.run.smithi097.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-25T16:28:38.902 INFO:teuthology.orchestra.run.smithi097.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-25T16:28:38.902 INFO:teuthology.orchestra.run.smithi097.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-25T16:28:38.902 INFO:teuthology.orchestra.run.smithi097.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-25T16:28:38.902 INFO:teuthology.orchestra.run.smithi097.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-25T16:28:38.902 INFO:teuthology.orchestra.run.smithi097.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-25T16:28:38.902 INFO:teuthology.orchestra.run.smithi097.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-25T16:28:38.902 INFO:teuthology.orchestra.run.smithi097.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-25T16:28:38.902 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T16:28:38.902 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-03-25T16:28:39.260 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install ceph-mgr-rook 2024-03-25T16:28:39.832 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:04:11 ago on Mon 25 Mar 2024 04:24:28 PM UTC. 2024-03-25T16:28:39.982 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-03-25T16:28:39.983 INFO:teuthology.orchestra.run.smithi097.stdout:======================================================================================== 2024-03-25T16:28:39.983 INFO:teuthology.orchestra.run.smithi097.stdout: Package Arch Version Repository Size 2024-03-25T16:28:39.983 INFO:teuthology.orchestra.run.smithi097.stdout:======================================================================================== 2024-03-25T16:28:39.983 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2024-03-25T16:28:39.983 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-25T16:28:39.983 INFO:teuthology.orchestra.run.smithi097.stdout:Installing dependencies: 2024-03-25T16:28:39.983 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-25T16:28:39.984 INFO:teuthology.orchestra.run.smithi097.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-25T16:28:39.984 INFO:teuthology.orchestra.run.smithi097.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-25T16:28:39.984 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-25T16:28:39.984 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-25T16:28:39.984 INFO:teuthology.orchestra.run.smithi097.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-25T16:28:39.984 INFO:teuthology.orchestra.run.smithi097.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-25T16:28:39.984 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-25T16:28:39.984 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-25T16:28:39.984 INFO:teuthology.orchestra.run.smithi097.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-25T16:28:39.984 INFO:teuthology.orchestra.run.smithi097.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-25T16:28:39.984 INFO:teuthology.orchestra.run.smithi097.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-25T16:28:39.984 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T16:28:39.984 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2024-03-25T16:28:39.984 INFO:teuthology.orchestra.run.smithi097.stdout:======================================================================================== 2024-03-25T16:28:39.985 INFO:teuthology.orchestra.run.smithi097.stdout:Install 13 Packages 2024-03-25T16:28:39.985 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T16:28:39.985 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 2.0 M 2024-03-25T16:28:39.985 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 26 M 2024-03-25T16:28:39.985 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2024-03-25T16:28:40.427 INFO:teuthology.orchestra.run.smithi097.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 166 kB/s | 24 kB 00:00 2024-03-25T16:28:40.452 INFO:teuthology.orchestra.run.smithi097.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 105 kB/s | 18 kB 00:00 2024-03-25T16:28:40.519 INFO:teuthology.orchestra.run.smithi097.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.6 MB/s | 174 kB 00:00 2024-03-25T16:28:40.553 INFO:teuthology.orchestra.run.smithi097.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.3 MB/s | 132 kB 00:00 2024-03-25T16:28:40.578 INFO:teuthology.orchestra.run.smithi097.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 3.5 MB/s | 211 kB 00:00 2024-03-25T16:28:40.604 INFO:teuthology.orchestra.run.smithi097.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 853 kB/s | 43 kB 00:00 2024-03-25T16:28:40.712 INFO:teuthology.orchestra.run.smithi097.stdout:(7/13): python3-certifi-2021.10.8-2.el9.noarch. 138 kB/s | 15 kB 00:00 2024-03-25T16:28:40.737 INFO:teuthology.orchestra.run.smithi097.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 202 kB/s | 32 kB 00:00 2024-03-25T16:28:40.929 INFO:teuthology.orchestra.run.smithi097.stdout:(9/13): python3-google-auth-2.28.2-1.el9.noarch 934 kB/s | 203 kB 00:00 2024-03-25T16:28:41.013 INFO:teuthology.orchestra.run.smithi097.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 708 kB/s | 59 kB 00:00 2024-03-25T16:28:41.038 INFO:teuthology.orchestra.run.smithi097.stdout:(11/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el 69 kB/s | 52 kB 00:00 2024-03-25T16:28:41.105 INFO:teuthology.orchestra.run.smithi097.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 976 kB/s | 90 kB 00:00 2024-03-25T16:28:41.356 INFO:teuthology.orchestra.run.smithi097.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 1.7 MB/s | 1.0 MB 00:00 2024-03-25T16:28:41.358 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:28:41.358 INFO:teuthology.orchestra.run.smithi097.stdout:Total 1.5 MB/s | 2.0 MB 00:01 2024-03-25T16:28:41.358 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2024-03-25T16:28:41.372 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2024-03-25T16:28:41.372 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2024-03-25T16:28:41.466 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2024-03-25T16:28:41.466 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2024-03-25T16:28:41.994 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2024-03-25T16:28:42.170 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-25T16:28:42.223 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-25T16:28:42.279 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-25T16:28:42.316 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-25T16:28:42.357 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-25T16:28:42.461 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-25T16:28:42.569 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-25T16:28:42.626 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-25T16:28:43.361 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-25T16:28:43.431 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-25T16:28:43.474 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-25T16:28:43.538 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-25T16:28:43.560 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T16:28:44.331 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T16:28:44.331 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-25T16:28:44.331 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-25T16:28:44.331 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-25T16:28:44.331 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-25T16:28:44.331 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-25T16:28:44.331 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-25T16:28:44.331 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-25T16:28:44.331 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-25T16:28:44.331 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-25T16:28:44.331 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-25T16:28:44.331 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-25T16:28:44.332 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-25T16:28:44.688 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-25T16:28:44.688 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T16:28:44.688 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2024-03-25T16:28:44.688 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:28:44.688 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-25T16:28:44.688 INFO:teuthology.orchestra.run.smithi097.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-25T16:28:44.689 INFO:teuthology.orchestra.run.smithi097.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-25T16:28:44.689 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-25T16:28:44.689 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-25T16:28:44.689 INFO:teuthology.orchestra.run.smithi097.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-25T16:28:44.689 INFO:teuthology.orchestra.run.smithi097.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-25T16:28:44.689 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-25T16:28:44.689 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-25T16:28:44.689 INFO:teuthology.orchestra.run.smithi097.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-25T16:28:44.689 INFO:teuthology.orchestra.run.smithi097.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-25T16:28:44.689 INFO:teuthology.orchestra.run.smithi097.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-25T16:28:44.689 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T16:28:44.689 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-03-25T16:28:44.964 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install ceph-mgr-cephadm 2024-03-25T16:28:45.521 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:04:17 ago on Mon 25 Mar 2024 04:24:28 PM UTC. 2024-03-25T16:28:45.671 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-03-25T16:28:45.672 INFO:teuthology.orchestra.run.smithi097.stdout:======================================================================================== 2024-03-25T16:28:45.672 INFO:teuthology.orchestra.run.smithi097.stdout: Package Arch Version Repository Size 2024-03-25T16:28:45.672 INFO:teuthology.orchestra.run.smithi097.stdout:======================================================================================== 2024-03-25T16:28:45.672 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2024-03-25T16:28:45.672 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-25T16:28:45.672 INFO:teuthology.orchestra.run.smithi097.stdout:Installing dependencies: 2024-03-25T16:28:45.672 INFO:teuthology.orchestra.run.smithi097.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-25T16:28:45.673 INFO:teuthology.orchestra.run.smithi097.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-25T16:28:45.673 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-25T16:28:45.673 INFO:teuthology.orchestra.run.smithi097.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-25T16:28:45.673 INFO:teuthology.orchestra.run.smithi097.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-25T16:28:45.673 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T16:28:45.673 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2024-03-25T16:28:45.673 INFO:teuthology.orchestra.run.smithi097.stdout:======================================================================================== 2024-03-25T16:28:45.673 INFO:teuthology.orchestra.run.smithi097.stdout:Install 6 Packages 2024-03-25T16:28:45.673 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T16:28:45.674 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 6.8 M 2024-03-25T16:28:45.674 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 33 M 2024-03-25T16:28:45.674 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2024-03-25T16:28:46.235 INFO:teuthology.orchestra.run.smithi097.stdout:(1/6): python3-babel-2.9.1-2.el9.noarch.rpm 18 MB/s | 5.8 MB 00:00 2024-03-25T16:28:46.302 INFO:teuthology.orchestra.run.smithi097.stdout:(2/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 640 kB/s | 248 kB 00:00 2024-03-25T16:28:46.669 INFO:teuthology.orchestra.run.smithi097.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 168 kB/s | 58 kB 00:00 2024-03-25T16:28:46.786 INFO:teuthology.orchestra.run.smithi097.stdout:(4/6): python3-typing-extensions-4.4.0-2.el9.no 437 kB/s | 51 kB 00:00 2024-03-25T16:28:46.811 INFO:teuthology.orchestra.run.smithi097.stdout:(5/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 954 kB/s | 548 kB 00:00 2024-03-25T16:28:46.836 INFO:teuthology.orchestra.run.smithi097.stdout:(6/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 150 kB/s | 138 kB 00:00 2024-03-25T16:28:46.839 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:28:46.839 INFO:teuthology.orchestra.run.smithi097.stdout:Total 5.8 MB/s | 6.8 MB 00:01 2024-03-25T16:28:46.852 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2024-03-25T16:28:46.862 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2024-03-25T16:28:46.862 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2024-03-25T16:28:46.950 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2024-03-25T16:28:46.950 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2024-03-25T16:28:47.248 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2024-03-25T16:28:47.429 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-25T16:28:47.479 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-25T16:28:48.018 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-25T16:28:48.098 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-25T16:28:48.161 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-25T16:28:48.184 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T16:28:48.992 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T16:28:48.992 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-25T16:28:48.992 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-25T16:28:48.992 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-25T16:28:48.992 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-25T16:28:48.992 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-25T16:28:49.284 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-25T16:28:49.285 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T16:28:49.285 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2024-03-25T16:28:49.285 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:28:49.285 INFO:teuthology.orchestra.run.smithi097.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-25T16:28:49.285 INFO:teuthology.orchestra.run.smithi097.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-25T16:28:49.285 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-25T16:28:49.285 INFO:teuthology.orchestra.run.smithi097.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-25T16:28:49.285 INFO:teuthology.orchestra.run.smithi097.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-25T16:28:49.285 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T16:28:49.285 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-03-25T16:28:49.524 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install ceph-fuse 2024-03-25T16:28:50.090 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:04:22 ago on Mon 25 Mar 2024 04:24:28 PM UTC. 2024-03-25T16:28:50.243 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-03-25T16:28:50.243 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-03-25T16:28:50.243 INFO:teuthology.orchestra.run.smithi097.stdout: Package Arch Version Repository Size 2024-03-25T16:28:50.243 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-03-25T16:28:50.244 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2024-03-25T16:28:50.244 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-25T16:28:50.244 INFO:teuthology.orchestra.run.smithi097.stdout:Installing dependencies: 2024-03-25T16:28:50.244 INFO:teuthology.orchestra.run.smithi097.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-25T16:28:50.244 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T16:28:50.244 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2024-03-25T16:28:50.245 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-03-25T16:28:50.245 INFO:teuthology.orchestra.run.smithi097.stdout:Install 2 Packages 2024-03-25T16:28:50.245 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T16:28:50.245 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 964 k 2024-03-25T16:28:50.245 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 2.8 M 2024-03-25T16:28:50.245 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2024-03-25T16:28:50.380 INFO:teuthology.orchestra.run.smithi097.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 572 kB/s | 79 kB 00:00 2024-03-25T16:28:51.622 INFO:teuthology.orchestra.run.smithi097.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 642 kB/s | 885 kB 00:01 2024-03-25T16:28:51.624 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:28:51.624 INFO:teuthology.orchestra.run.smithi097.stdout:Total 697 kB/s | 964 kB 00:01 2024-03-25T16:28:51.624 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2024-03-25T16:28:51.647 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2024-03-25T16:28:51.647 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2024-03-25T16:28:51.723 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2024-03-25T16:28:51.723 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2024-03-25T16:28:51.902 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2024-03-25T16:28:52.014 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-25T16:28:52.080 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:28:52.831 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:28:52.831 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T16:28:53.164 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-25T16:28:53.164 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T16:28:53.164 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2024-03-25T16:28:53.164 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-25T16:28:53.165 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T16:28:53.165 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-03-25T16:28:53.464 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install ceph-volume 2024-03-25T16:28:54.039 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:04:26 ago on Mon 25 Mar 2024 04:24:28 PM UTC. 2024-03-25T16:28:54.187 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-03-25T16:28:54.188 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-03-25T16:28:54.188 INFO:teuthology.orchestra.run.smithi097.stdout: Package Arch Version Repository Size 2024-03-25T16:28:54.188 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-03-25T16:28:54.188 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2024-03-25T16:28:54.188 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-25T16:28:54.188 INFO:teuthology.orchestra.run.smithi097.stdout:Installing dependencies: 2024-03-25T16:28:54.188 INFO:teuthology.orchestra.run.smithi097.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-25T16:28:54.188 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T16:28:54.189 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2024-03-25T16:28:54.189 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-03-25T16:28:54.189 INFO:teuthology.orchestra.run.smithi097.stdout:Install 2 Packages 2024-03-25T16:28:54.189 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T16:28:54.189 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 332 k 2024-03-25T16:28:54.189 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 1.5 M 2024-03-25T16:28:54.189 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2024-03-25T16:28:54.294 INFO:teuthology.orchestra.run.smithi097.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 662 kB/s | 69 kB 00:00 2024-03-25T16:28:55.585 INFO:teuthology.orchestra.run.smithi097.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 189 kB/s | 264 kB 00:01 2024-03-25T16:28:55.587 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:28:55.587 INFO:teuthology.orchestra.run.smithi097.stdout:Total 238 kB/s | 332 kB 00:01 2024-03-25T16:28:55.587 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2024-03-25T16:28:55.597 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2024-03-25T16:28:55.597 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2024-03-25T16:28:55.645 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2024-03-25T16:28:55.646 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2024-03-25T16:28:55.794 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2024-03-25T16:28:55.976 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-25T16:28:56.013 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T16:28:56.028 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T16:28:56.029 INFO:teuthology.orchestra.run.smithi097.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:28:56.029 INFO:teuthology.orchestra.run.smithi097.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-25T16:28:56.029 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T16:28:56.741 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-25T16:28:57.069 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-25T16:28:57.070 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T16:28:57.070 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2024-03-25T16:28:57.070 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:28:57.070 INFO:teuthology.orchestra.run.smithi097.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-25T16:28:57.070 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T16:28:57.070 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-03-25T16:28:57.409 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install librados-devel 2024-03-25T16:28:57.986 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:04:29 ago on Mon 25 Mar 2024 04:24:28 PM UTC. 2024-03-25T16:28:58.135 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-03-25T16:28:58.136 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-03-25T16:28:58.136 INFO:teuthology.orchestra.run.smithi097.stdout: Package Arch Version Repo Size 2024-03-25T16:28:58.136 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-03-25T16:28:58.136 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2024-03-25T16:28:58.136 INFO:teuthology.orchestra.run.smithi097.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-25T16:28:58.137 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T16:28:58.137 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2024-03-25T16:28:58.137 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-03-25T16:28:58.137 INFO:teuthology.orchestra.run.smithi097.stdout:Install 1 Package 2024-03-25T16:28:58.137 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T16:28:58.137 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 127 k 2024-03-25T16:28:58.137 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 456 k 2024-03-25T16:28:58.137 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2024-03-25T16:28:58.750 INFO:teuthology.orchestra.run.smithi097.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 207 kB/s | 127 kB 00:00 2024-03-25T16:28:58.751 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:28:58.751 INFO:teuthology.orchestra.run.smithi097.stdout:Total 206 kB/s | 127 kB 00:00 2024-03-25T16:28:58.751 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2024-03-25T16:28:58.757 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2024-03-25T16:28:58.757 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2024-03-25T16:28:58.814 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2024-03-25T16:28:58.814 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2024-03-25T16:28:58.960 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2024-03-25T16:28:59.050 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:28:59.561 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:28:59.876 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:28:59.876 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T16:28:59.876 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2024-03-25T16:28:59.876 INFO:teuthology.orchestra.run.smithi097.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:28:59.876 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T16:28:59.877 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-03-25T16:29:00.094 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install libcephfs2 2024-03-25T16:29:00.666 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:04:32 ago on Mon 25 Mar 2024 04:24:28 PM UTC. 2024-03-25T16:29:00.765 INFO:teuthology.orchestra.run.smithi097.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:29:00.820 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-03-25T16:29:00.821 INFO:teuthology.orchestra.run.smithi097.stdout:Nothing to do. 2024-03-25T16:29:00.821 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-03-25T16:29:00.888 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install libcephfs-devel 2024-03-25T16:29:01.453 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:04:33 ago on Mon 25 Mar 2024 04:24:28 PM UTC. 2024-03-25T16:29:01.603 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-03-25T16:29:01.604 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-03-25T16:29:01.604 INFO:teuthology.orchestra.run.smithi097.stdout: Package Arch Version Repo Size 2024-03-25T16:29:01.604 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-03-25T16:29:01.604 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2024-03-25T16:29:01.604 INFO:teuthology.orchestra.run.smithi097.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-25T16:29:01.604 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T16:29:01.605 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2024-03-25T16:29:01.605 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-03-25T16:29:01.605 INFO:teuthology.orchestra.run.smithi097.stdout:Install 1 Package 2024-03-25T16:29:01.605 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T16:29:01.605 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 31 k 2024-03-25T16:29:01.605 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 140 k 2024-03-25T16:29:01.605 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2024-03-25T16:29:02.238 INFO:teuthology.orchestra.run.smithi097.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 49 kB/s | 31 kB 00:00 2024-03-25T16:29:02.238 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:29:02.239 INFO:teuthology.orchestra.run.smithi097.stdout:Total 49 kB/s | 31 kB 00:00 2024-03-25T16:29:02.239 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2024-03-25T16:29:02.243 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2024-03-25T16:29:02.243 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2024-03-25T16:29:02.271 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2024-03-25T16:29:02.271 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2024-03-25T16:29:02.398 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2024-03-25T16:29:02.503 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:29:02.871 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:29:03.135 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:29:03.135 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T16:29:03.135 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2024-03-25T16:29:03.135 INFO:teuthology.orchestra.run.smithi097.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:29:03.135 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T16:29:03.135 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-03-25T16:29:03.294 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install librados2 2024-03-25T16:29:03.860 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:04:35 ago on Mon 25 Mar 2024 04:24:28 PM UTC. 2024-03-25T16:29:03.957 INFO:teuthology.orchestra.run.smithi097.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:29:04.008 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-03-25T16:29:04.008 INFO:teuthology.orchestra.run.smithi097.stdout:Nothing to do. 2024-03-25T16:29:04.009 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-03-25T16:29:04.056 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install librbd1 2024-03-25T16:29:04.622 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:04:36 ago on Mon 25 Mar 2024 04:24:28 PM UTC. 2024-03-25T16:29:04.719 INFO:teuthology.orchestra.run.smithi097.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:29:04.770 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-03-25T16:29:04.771 INFO:teuthology.orchestra.run.smithi097.stdout:Nothing to do. 2024-03-25T16:29:04.771 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-03-25T16:29:04.831 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install python3-rados 2024-03-25T16:29:05.388 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:04:37 ago on Mon 25 Mar 2024 04:24:28 PM UTC. 2024-03-25T16:29:05.483 INFO:teuthology.orchestra.run.smithi097.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:29:05.536 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-03-25T16:29:05.537 INFO:teuthology.orchestra.run.smithi097.stdout:Nothing to do. 2024-03-25T16:29:05.537 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-03-25T16:29:05.592 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install python3-rgw 2024-03-25T16:29:06.163 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:04:38 ago on Mon 25 Mar 2024 04:24:28 PM UTC. 2024-03-25T16:29:06.266 INFO:teuthology.orchestra.run.smithi097.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:29:06.322 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-03-25T16:29:06.322 INFO:teuthology.orchestra.run.smithi097.stdout:Nothing to do. 2024-03-25T16:29:06.322 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-03-25T16:29:06.376 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install python3-cephfs 2024-03-25T16:29:06.959 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:04:38 ago on Mon 25 Mar 2024 04:24:28 PM UTC. 2024-03-25T16:29:07.060 INFO:teuthology.orchestra.run.smithi097.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:29:07.113 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-03-25T16:29:07.114 INFO:teuthology.orchestra.run.smithi097.stdout:Nothing to do. 2024-03-25T16:29:07.114 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-03-25T16:29:07.165 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install python3-rbd 2024-03-25T16:29:07.731 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:04:39 ago on Mon 25 Mar 2024 04:24:28 PM UTC. 2024-03-25T16:29:07.827 INFO:teuthology.orchestra.run.smithi097.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:29:07.878 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-03-25T16:29:07.879 INFO:teuthology.orchestra.run.smithi097.stdout:Nothing to do. 2024-03-25T16:29:07.879 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-03-25T16:29:07.930 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install rbd-fuse 2024-03-25T16:29:08.495 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:04:40 ago on Mon 25 Mar 2024 04:24:28 PM UTC. 2024-03-25T16:29:08.644 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-03-25T16:29:08.645 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-03-25T16:29:08.645 INFO:teuthology.orchestra.run.smithi097.stdout: Package Architecture Version Repository Size 2024-03-25T16:29:08.645 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-03-25T16:29:08.645 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2024-03-25T16:29:08.645 INFO:teuthology.orchestra.run.smithi097.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-25T16:29:08.645 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T16:29:08.645 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2024-03-25T16:29:08.645 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-03-25T16:29:08.645 INFO:teuthology.orchestra.run.smithi097.stdout:Install 1 Package 2024-03-25T16:29:08.645 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T16:29:08.646 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 85 k 2024-03-25T16:29:08.646 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 231 k 2024-03-25T16:29:08.646 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2024-03-25T16:29:09.508 INFO:teuthology.orchestra.run.smithi097.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 99 kB/s | 85 kB 00:00 2024-03-25T16:29:09.508 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:29:09.509 INFO:teuthology.orchestra.run.smithi097.stdout:Total 99 kB/s | 85 kB 00:00 2024-03-25T16:29:09.509 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2024-03-25T16:29:09.517 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2024-03-25T16:29:09.517 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2024-03-25T16:29:09.576 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2024-03-25T16:29:09.577 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2024-03-25T16:29:09.715 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2024-03-25T16:29:09.806 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:29:10.243 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:29:10.535 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:29:10.535 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T16:29:10.535 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2024-03-25T16:29:10.535 INFO:teuthology.orchestra.run.smithi097.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:29:10.535 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T16:29:10.535 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-03-25T16:29:10.793 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install rbd-mirror 2024-03-25T16:29:11.358 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:04:43 ago on Mon 25 Mar 2024 04:24:28 PM UTC. 2024-03-25T16:29:11.509 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-03-25T16:29:11.510 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-03-25T16:29:11.510 INFO:teuthology.orchestra.run.smithi097.stdout: Package Arch Version Repo Size 2024-03-25T16:29:11.510 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-03-25T16:29:11.510 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2024-03-25T16:29:11.510 INFO:teuthology.orchestra.run.smithi097.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T16:29:11.510 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T16:29:11.510 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2024-03-25T16:29:11.510 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-03-25T16:29:11.510 INFO:teuthology.orchestra.run.smithi097.stdout:Install 1 Package 2024-03-25T16:29:11.510 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T16:29:11.511 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 3.1 M 2024-03-25T16:29:11.511 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 13 M 2024-03-25T16:29:11.511 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2024-03-25T16:29:14.778 INFO:teuthology.orchestra.run.smithi097.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 968 kB/s | 3.1 MB 00:03 2024-03-25T16:29:14.779 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:29:14.779 INFO:teuthology.orchestra.run.smithi097.stdout:Total 967 kB/s | 3.1 MB 00:03 2024-03-25T16:29:14.779 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2024-03-25T16:29:14.790 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2024-03-25T16:29:14.790 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2024-03-25T16:29:14.866 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2024-03-25T16:29:14.866 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2024-03-25T16:29:15.300 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2024-03-25T16:29:15.354 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:29:15.378 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:29:15.378 INFO:teuthology.orchestra.run.smithi097.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:29:15.378 INFO:teuthology.orchestra.run.smithi097.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-25T16:29:15.378 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T16:29:15.378 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T16:29:15.378 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T16:29:16.482 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:29:16.482 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T16:29:16.482 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2024-03-25T16:29:16.482 INFO:teuthology.orchestra.run.smithi097.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:29:16.482 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T16:29:16.482 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-03-25T16:29:16.756 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install rbd-nbd 2024-03-25T16:29:17.348 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:04:49 ago on Mon 25 Mar 2024 04:24:28 PM UTC. 2024-03-25T16:29:17.497 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-03-25T16:29:17.499 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-03-25T16:29:17.499 INFO:teuthology.orchestra.run.smithi097.stdout: Package Architecture Version Repository Size 2024-03-25T16:29:17.499 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-03-25T16:29:17.499 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2024-03-25T16:29:17.499 INFO:teuthology.orchestra.run.smithi097.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-25T16:29:17.500 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T16:29:17.500 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2024-03-25T16:29:17.500 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-03-25T16:29:17.500 INFO:teuthology.orchestra.run.smithi097.stdout:Install 1 Package 2024-03-25T16:29:17.500 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T16:29:17.500 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 171 k 2024-03-25T16:29:17.501 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 494 k 2024-03-25T16:29:17.501 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2024-03-25T16:29:18.240 INFO:teuthology.orchestra.run.smithi097.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 231 kB/s | 171 kB 00:00 2024-03-25T16:29:18.241 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:29:18.241 INFO:teuthology.orchestra.run.smithi097.stdout:Total 231 kB/s | 171 kB 00:00 2024-03-25T16:29:18.241 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2024-03-25T16:29:18.249 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2024-03-25T16:29:18.249 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2024-03-25T16:29:18.308 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2024-03-25T16:29:18.308 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2024-03-25T16:29:18.457 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2024-03-25T16:29:18.571 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:29:19.017 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:29:19.308 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:29:19.309 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T16:29:19.309 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2024-03-25T16:29:19.309 INFO:teuthology.orchestra.run.smithi097.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:29:19.309 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T16:29:19.309 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-03-25T16:29:19.541 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install ceph-base-debuginfo 2024-03-25T16:29:20.112 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:04:52 ago on Mon 25 Mar 2024 04:24:28 PM UTC. 2024-03-25T16:29:20.265 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-03-25T16:29:20.265 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-03-25T16:29:20.265 INFO:teuthology.orchestra.run.smithi097.stdout: Package Arch Version Repo Size 2024-03-25T16:29:20.265 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-03-25T16:29:20.265 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2024-03-25T16:29:20.265 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-base-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 68 M 2024-03-25T16:29:20.265 INFO:teuthology.orchestra.run.smithi097.stdout:Installing dependencies: 2024-03-25T16:29:20.265 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 22 M 2024-03-25T16:29:20.265 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T16:29:20.266 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2024-03-25T16:29:20.266 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-03-25T16:29:20.266 INFO:teuthology.orchestra.run.smithi097.stdout:Install 2 Packages 2024-03-25T16:29:20.266 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T16:29:20.266 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 91 M 2024-03-25T16:29:20.266 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 498 M 2024-03-25T16:29:20.266 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2024-03-25T16:30:10.439 INFO:teuthology.orchestra.run.smithi097.stdout:(1/2): ceph-debuginfo-19.0.0-1439.g5dcb190d.el9 458 kB/s | 22 MB 00:50 2024-03-25T16:31:27.257 INFO:teuthology.orchestra.run.smithi097.stdout:(2/2): ceph-base-debuginfo-19.0.0-1439.g5dcb190 549 kB/s | 68 MB 02:06 2024-03-25T16:31:27.257 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:31:27.258 INFO:teuthology.orchestra.run.smithi097.stdout:Total 730 kB/s | 91 MB 02:06 2024-03-25T16:31:27.258 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2024-03-25T16:31:27.263 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2024-03-25T16:31:27.263 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2024-03-25T16:31:27.846 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2024-03-25T16:31:27.846 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2024-03-25T16:31:30.938 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2024-03-25T16:31:38.041 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T16:31:38.116 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T16:31:41.444 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T16:31:41.444 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/2 2024-03-25T16:31:41.706 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:31:41.706 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T16:31:41.706 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2024-03-25T16:31:41.706 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:31:41.706 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:31:41.706 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T16:31:41.706 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-03-25T16:31:41.933 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install ceph-common-debuginfo 2024-03-25T16:31:42.494 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:07:14 ago on Mon 25 Mar 2024 04:24:28 PM UTC. 2024-03-25T16:31:42.643 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-03-25T16:31:42.643 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-03-25T16:31:42.643 INFO:teuthology.orchestra.run.smithi097.stdout: Package Arch Version Repo Size 2024-03-25T16:31:42.643 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-03-25T16:31:42.643 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2024-03-25T16:31:42.643 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-common-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 416 M 2024-03-25T16:31:42.643 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T16:31:42.643 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2024-03-25T16:31:42.644 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-03-25T16:31:42.644 INFO:teuthology.orchestra.run.smithi097.stdout:Install 1 Package 2024-03-25T16:31:42.644 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T16:31:42.644 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 416 M 2024-03-25T16:31:42.644 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 1.7 G 2024-03-25T16:31:42.644 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2024-03-25T16:41:59.060 INFO:teuthology.orchestra.run.smithi098.stdout:ceph-common-debuginfo-19.0.0-1439.g5dcb190d.el9 449 kB/s | 416 MB 15:49 2024-03-25T16:41:59.081 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:41:59.081 INFO:teuthology.orchestra.run.smithi098.stdout:Total 449 kB/s | 416 MB 15:49 2024-03-25T16:41:59.081 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-03-25T16:41:59.082 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-03-25T16:41:59.082 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-03-25T16:42:01.541 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-03-25T16:42:01.541 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-03-25T16:42:43.892 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-03-25T16:42:44.188 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T16:42:52.404 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T16:42:52.664 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T16:42:52.664 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T16:42:52.664 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-03-25T16:42:52.664 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:42:52.664 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T16:42:52.664 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-25T16:42:52.929 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2024-03-25T16:42:53.497 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:22:48 ago on Mon 25 Mar 2024 04:20:05 PM UTC. 2024-03-25T16:42:53.646 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-25T16:42:53.647 INFO:teuthology.orchestra.run.smithi098.stdout:======================================================================================== 2024-03-25T16:42:53.647 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repo Size 2024-03-25T16:42:53.647 INFO:teuthology.orchestra.run.smithi098.stdout:======================================================================================== 2024-03-25T16:42:53.647 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-03-25T16:42:53.647 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.4 M 2024-03-25T16:42:53.647 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T16:42:53.647 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-03-25T16:42:53.647 INFO:teuthology.orchestra.run.smithi098.stdout:======================================================================================== 2024-03-25T16:42:53.647 INFO:teuthology.orchestra.run.smithi098.stdout:Install 1 Package 2024-03-25T16:42:53.647 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T16:42:53.647 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 1.4 M 2024-03-25T16:42:53.648 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 6.4 M 2024-03-25T16:42:53.648 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-03-25T16:43:00.716 INFO:teuthology.orchestra.run.smithi098.stdout:ceph-immutable-object-cache-debuginfo-19.0.0-14 207 kB/s | 1.4 MB 00:07 2024-03-25T16:43:00.717 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:43:00.717 INFO:teuthology.orchestra.run.smithi098.stdout:Total 207 kB/s | 1.4 MB 00:07 2024-03-25T16:43:00.717 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-03-25T16:43:00.721 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-03-25T16:43:00.721 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-03-25T16:43:00.765 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-03-25T16:43:00.766 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-03-25T16:43:01.024 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-03-25T16:43:01.122 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T16:43:01.539 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T16:43:01.819 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T16:43:01.820 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T16:43:01.820 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-03-25T16:43:01.820 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:43:01.820 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T16:43:01.820 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-25T16:43:02.014 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph-radosgw-debuginfo 2024-03-25T16:43:02.578 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:22:57 ago on Mon 25 Mar 2024 04:20:05 PM UTC. 2024-03-25T16:43:02.726 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-25T16:43:02.727 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T16:43:02.727 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repo Size 2024-03-25T16:43:02.727 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T16:43:02.727 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-03-25T16:43:02.727 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-radosgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 176 M 2024-03-25T16:43:02.727 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T16:43:02.728 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-03-25T16:43:02.728 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T16:43:02.728 INFO:teuthology.orchestra.run.smithi098.stdout:Install 1 Package 2024-03-25T16:43:02.728 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T16:43:02.728 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 176 M 2024-03-25T16:43:02.728 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 753 M 2024-03-25T16:43:02.728 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-03-25T16:51:18.876 INFO:teuthology.orchestra.run.smithi098.stdout:ceph-radosgw-debuginfo-19.0.0-1439.g5dcb190d.el 363 kB/s | 176 MB 08:16 2024-03-25T16:51:18.877 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:51:18.877 INFO:teuthology.orchestra.run.smithi098.stdout:Total 363 kB/s | 176 MB 08:16 2024-03-25T16:51:18.877 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-03-25T16:51:18.878 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-03-25T16:51:18.878 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-03-25T16:51:19.939 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-03-25T16:51:19.939 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-03-25T16:51:38.154 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-03-25T16:51:38.244 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T16:51:43.094 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T16:51:43.359 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T16:51:43.359 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T16:51:43.359 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-03-25T16:51:43.359 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:51:43.359 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T16:51:43.359 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-25T16:51:43.567 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph-test-debuginfo 2024-03-25T16:51:44.129 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:31:39 ago on Mon 25 Mar 2024 04:20:05 PM UTC. 2024-03-25T16:51:44.277 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-25T16:51:44.278 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T16:51:44.278 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repo Size 2024-03-25T16:51:44.278 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T16:51:44.279 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-03-25T16:51:44.279 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-test-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 672 M 2024-03-25T16:51:44.279 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T16:51:44.279 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-03-25T16:51:44.279 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T16:51:44.279 INFO:teuthology.orchestra.run.smithi098.stdout:Install 1 Package 2024-03-25T16:51:44.279 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T16:51:44.279 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 672 M 2024-03-25T16:51:44.279 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 3.1 G 2024-03-25T16:51:44.279 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-03-25T16:55:17.953 INFO:teuthology.orchestra.run.smithi097.stdout:ceph-common-debuginfo-19.0.0-1439.g5dcb190d.el9 301 kB/s | 416 MB 23:35 2024-03-25T16:55:17.954 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:55:17.954 INFO:teuthology.orchestra.run.smithi097.stdout:Total 301 kB/s | 416 MB 23:35 2024-03-25T16:55:17.954 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2024-03-25T16:55:17.957 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2024-03-25T16:55:17.957 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2024-03-25T16:55:20.429 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2024-03-25T16:55:20.430 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2024-03-25T16:56:02.519 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2024-03-25T16:56:02.614 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T16:56:09.571 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T16:56:09.826 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T16:56:09.827 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T16:56:09.827 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2024-03-25T16:56:09.827 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:56:09.827 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T16:56:09.827 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-03-25T16:56:10.097 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2024-03-25T16:56:10.660 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:31:42 ago on Mon 25 Mar 2024 04:24:28 PM UTC. 2024-03-25T16:56:10.808 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-03-25T16:56:10.808 INFO:teuthology.orchestra.run.smithi097.stdout:======================================================================================== 2024-03-25T16:56:10.808 INFO:teuthology.orchestra.run.smithi097.stdout: Package Arch Version Repo Size 2024-03-25T16:56:10.809 INFO:teuthology.orchestra.run.smithi097.stdout:======================================================================================== 2024-03-25T16:56:10.809 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2024-03-25T16:56:10.809 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.4 M 2024-03-25T16:56:10.809 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T16:56:10.809 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2024-03-25T16:56:10.809 INFO:teuthology.orchestra.run.smithi097.stdout:======================================================================================== 2024-03-25T16:56:10.809 INFO:teuthology.orchestra.run.smithi097.stdout:Install 1 Package 2024-03-25T16:56:10.809 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T16:56:10.810 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 1.4 M 2024-03-25T16:56:10.810 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 6.4 M 2024-03-25T16:56:10.810 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2024-03-25T16:56:19.001 INFO:teuthology.orchestra.run.smithi097.stdout:ceph-immutable-object-cache-debuginfo-19.0.0-14 178 kB/s | 1.4 MB 00:08 2024-03-25T16:56:19.002 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:56:19.002 INFO:teuthology.orchestra.run.smithi097.stdout:Total 178 kB/s | 1.4 MB 00:08 2024-03-25T16:56:19.003 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2024-03-25T16:56:19.007 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2024-03-25T16:56:19.007 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2024-03-25T16:56:19.051 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2024-03-25T16:56:19.051 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2024-03-25T16:56:19.308 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2024-03-25T16:56:19.408 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T16:56:19.807 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T16:56:20.071 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T16:56:20.071 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T16:56:20.072 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2024-03-25T16:56:20.072 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:56:20.072 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T16:56:20.072 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-03-25T16:56:20.278 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install ceph-radosgw-debuginfo 2024-03-25T16:56:20.840 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:31:52 ago on Mon 25 Mar 2024 04:24:28 PM UTC. 2024-03-25T16:56:20.989 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-03-25T16:56:20.990 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-03-25T16:56:20.990 INFO:teuthology.orchestra.run.smithi097.stdout: Package Arch Version Repo Size 2024-03-25T16:56:20.990 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-03-25T16:56:20.990 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2024-03-25T16:56:20.990 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-radosgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 176 M 2024-03-25T16:56:20.991 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T16:56:20.991 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2024-03-25T16:56:20.991 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-03-25T16:56:20.991 INFO:teuthology.orchestra.run.smithi097.stdout:Install 1 Package 2024-03-25T16:56:20.991 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T16:56:20.991 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 176 M 2024-03-25T16:56:20.991 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 753 M 2024-03-25T16:56:20.991 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2024-03-25T17:03:11.178 INFO:teuthology.orchestra.run.smithi097.stdout:ceph-radosgw-debuginfo-19.0.0-1439.g5dcb190d.el 439 kB/s | 176 MB 06:50 2024-03-25T17:03:11.178 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:03:11.178 INFO:teuthology.orchestra.run.smithi097.stdout:Total 439 kB/s | 176 MB 06:50 2024-03-25T17:03:11.179 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2024-03-25T17:03:11.182 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2024-03-25T17:03:11.182 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2024-03-25T17:03:12.239 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2024-03-25T17:03:12.239 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2024-03-25T17:03:30.437 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2024-03-25T17:03:30.599 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T17:03:34.983 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T17:03:35.240 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T17:03:35.240 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T17:03:35.240 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2024-03-25T17:03:35.240 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:03:35.240 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T17:03:35.240 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-03-25T17:03:35.471 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install ceph-test-debuginfo 2024-03-25T17:03:36.032 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:39:08 ago on Mon 25 Mar 2024 04:24:28 PM UTC. 2024-03-25T17:03:36.181 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-03-25T17:03:36.181 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-03-25T17:03:36.181 INFO:teuthology.orchestra.run.smithi097.stdout: Package Arch Version Repo Size 2024-03-25T17:03:36.182 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-03-25T17:03:36.182 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2024-03-25T17:03:36.182 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-test-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 672 M 2024-03-25T17:03:36.182 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T17:03:36.182 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2024-03-25T17:03:36.182 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-03-25T17:03:36.182 INFO:teuthology.orchestra.run.smithi097.stdout:Install 1 Package 2024-03-25T17:03:36.182 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T17:03:36.183 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 672 M 2024-03-25T17:03:36.183 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 3.1 G 2024-03-25T17:03:36.183 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2024-03-25T17:29:00.334 INFO:teuthology.orchestra.run.smithi098.stdout:ceph-test-debuginfo-19.0.0-1439.g5dcb190d.el9.x 308 kB/s | 672 MB 37:16 2024-03-25T17:29:00.354 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:29:00.354 INFO:teuthology.orchestra.run.smithi098.stdout:Total 308 kB/s | 672 MB 37:16 2024-03-25T17:29:00.354 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-03-25T17:29:00.354 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-03-25T17:29:00.354 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-03-25T17:29:04.299 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-03-25T17:29:04.299 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-03-25T17:30:15.157 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-03-25T17:30:15.714 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:30:24.968 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:30:25.234 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:30:25.235 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T17:30:25.235 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-03-25T17:30:25.235 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:30:25.235 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T17:30:25.235 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-25T17:30:25.544 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph-base-debuginfo 2024-03-25T17:30:26.106 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 1:10:21 ago on Mon 25 Mar 2024 04:20:05 PM UTC. 2024-03-25T17:30:26.202 INFO:teuthology.orchestra.run.smithi098.stdout:Package ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T17:30:26.254 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-25T17:30:26.255 INFO:teuthology.orchestra.run.smithi098.stdout:Nothing to do. 2024-03-25T17:30:26.255 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-25T17:30:26.330 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph-mgr-debuginfo 2024-03-25T17:30:26.891 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 1:10:21 ago on Mon 25 Mar 2024 04:20:05 PM UTC. 2024-03-25T17:30:27.039 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-25T17:30:27.039 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T17:30:27.039 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repo Size 2024-03-25T17:30:27.039 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T17:30:27.039 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-03-25T17:30:27.040 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T17:30:27.040 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T17:30:27.040 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-03-25T17:30:27.040 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T17:30:27.040 INFO:teuthology.orchestra.run.smithi098.stdout:Install 1 Package 2024-03-25T17:30:27.040 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T17:30:27.040 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 20 M 2024-03-25T17:30:27.041 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 103 M 2024-03-25T17:30:27.041 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-03-25T17:31:26.760 INFO:teuthology.orchestra.run.smithi098.stdout:ceph-mgr-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 338 kB/s | 20 MB 00:59 2024-03-25T17:31:26.760 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:31:26.761 INFO:teuthology.orchestra.run.smithi098.stdout:Total 338 kB/s | 20 MB 00:59 2024-03-25T17:31:26.761 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-03-25T17:31:26.765 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-03-25T17:31:26.765 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-03-25T17:31:26.914 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-03-25T17:31:26.914 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-03-25T17:31:29.143 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-03-25T17:31:29.325 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:31:30.265 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:31:30.532 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:31:30.533 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T17:31:30.533 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-03-25T17:31:30.533 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:31:30.533 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T17:31:30.533 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-25T17:31:30.704 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph-mds-debuginfo 2024-03-25T17:31:31.269 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 1:11:26 ago on Mon 25 Mar 2024 04:20:05 PM UTC. 2024-03-25T17:31:31.416 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-25T17:31:31.417 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T17:31:31.417 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repo Size 2024-03-25T17:31:31.417 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T17:31:31.417 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-03-25T17:31:31.418 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mds-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 48 M 2024-03-25T17:31:31.418 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T17:31:31.418 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-03-25T17:31:31.418 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T17:31:31.418 INFO:teuthology.orchestra.run.smithi098.stdout:Install 1 Package 2024-03-25T17:31:31.418 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T17:31:31.418 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 48 M 2024-03-25T17:31:31.418 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 173 M 2024-03-25T17:31:31.418 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-03-25T17:33:02.744 INFO:teuthology.orchestra.run.smithi098.stdout:ceph-mds-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 535 kB/s | 48 MB 01:31 2024-03-25T17:33:02.745 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:33:02.745 INFO:teuthology.orchestra.run.smithi098.stdout:Total 535 kB/s | 48 MB 01:31 2024-03-25T17:33:02.745 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-03-25T17:33:02.749 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-03-25T17:33:02.749 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-03-25T17:33:03.060 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-03-25T17:33:03.060 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-03-25T17:33:07.784 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-03-25T17:33:07.926 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:33:09.326 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:33:09.595 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:33:09.596 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T17:33:09.596 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-03-25T17:33:09.596 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:33:09.596 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T17:33:09.596 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-25T17:33:09.774 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph-mon-debuginfo 2024-03-25T17:33:10.339 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 1:13:05 ago on Mon 25 Mar 2024 04:20:05 PM UTC. 2024-03-25T17:33:10.486 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-25T17:33:10.487 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T17:33:10.487 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repo Size 2024-03-25T17:33:10.487 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T17:33:10.487 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-03-25T17:33:10.487 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mon-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 M 2024-03-25T17:33:10.487 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T17:33:10.488 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-03-25T17:33:10.488 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T17:33:10.488 INFO:teuthology.orchestra.run.smithi098.stdout:Install 1 Package 2024-03-25T17:33:10.488 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T17:33:10.488 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 99 M 2024-03-25T17:33:10.488 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 449 M 2024-03-25T17:33:10.488 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-03-25T17:37:37.276 INFO:teuthology.orchestra.run.smithi098.stdout:ceph-mon-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 378 kB/s | 99 MB 04:26 2024-03-25T17:37:37.277 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:37:37.277 INFO:teuthology.orchestra.run.smithi098.stdout:Total 378 kB/s | 99 MB 04:26 2024-03-25T17:37:37.277 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-03-25T17:37:37.280 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-03-25T17:37:37.280 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-03-25T17:37:37.889 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-03-25T17:37:37.889 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-03-25T17:37:48.125 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-03-25T17:37:48.215 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:37:51.166 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:37:51.433 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:37:51.434 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T17:37:51.434 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-03-25T17:37:51.434 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:37:51.434 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T17:37:51.434 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-25T17:37:51.636 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph-osd-debuginfo 2024-03-25T17:37:52.200 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 1:17:47 ago on Mon 25 Mar 2024 04:20:05 PM UTC. 2024-03-25T17:37:52.348 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-25T17:37:52.348 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T17:37:52.349 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repo Size 2024-03-25T17:37:52.349 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T17:37:52.349 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-03-25T17:37:52.349 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-osd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 287 M 2024-03-25T17:37:52.349 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T17:37:52.349 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-03-25T17:37:52.349 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T17:37:52.349 INFO:teuthology.orchestra.run.smithi098.stdout:Install 1 Package 2024-03-25T17:37:52.349 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T17:37:52.350 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 287 M 2024-03-25T17:37:52.350 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 1.2 G 2024-03-25T17:37:52.350 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-03-25T17:44:11.681 INFO:teuthology.orchestra.run.smithi097.stdout:ceph-test-debuginfo-19.0.0-1439.g5dcb190d.el9.x 283 kB/s | 672 MB 40:35 2024-03-25T17:44:11.682 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:44:11.682 INFO:teuthology.orchestra.run.smithi097.stdout:Total 283 kB/s | 672 MB 40:35 2024-03-25T17:44:11.682 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2024-03-25T17:44:11.686 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2024-03-25T17:44:11.686 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2024-03-25T17:44:15.638 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2024-03-25T17:44:15.638 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2024-03-25T17:45:26.189 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2024-03-25T17:45:26.382 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:45:36.229 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:45:36.478 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:45:36.478 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T17:45:36.478 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2024-03-25T17:45:36.479 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:45:36.479 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T17:45:36.479 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-03-25T17:45:36.778 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install ceph-base-debuginfo 2024-03-25T17:45:37.340 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 1:21:09 ago on Mon 25 Mar 2024 04:24:28 PM UTC. 2024-03-25T17:45:37.437 INFO:teuthology.orchestra.run.smithi097.stdout:Package ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T17:45:37.489 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-03-25T17:45:37.490 INFO:teuthology.orchestra.run.smithi097.stdout:Nothing to do. 2024-03-25T17:45:37.490 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-03-25T17:45:37.550 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install ceph-mgr-debuginfo 2024-03-25T17:45:38.108 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 1:21:10 ago on Mon 25 Mar 2024 04:24:28 PM UTC. 2024-03-25T17:45:38.256 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-03-25T17:45:38.257 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-03-25T17:45:38.257 INFO:teuthology.orchestra.run.smithi097.stdout: Package Arch Version Repo Size 2024-03-25T17:45:38.257 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-03-25T17:45:38.257 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2024-03-25T17:45:38.257 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mgr-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T17:45:38.257 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T17:45:38.257 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2024-03-25T17:45:38.257 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-03-25T17:45:38.257 INFO:teuthology.orchestra.run.smithi097.stdout:Install 1 Package 2024-03-25T17:45:38.258 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T17:45:38.258 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 20 M 2024-03-25T17:45:38.258 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 103 M 2024-03-25T17:45:38.258 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2024-03-25T17:46:20.715 INFO:teuthology.orchestra.run.smithi097.stdout:ceph-mgr-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 475 kB/s | 20 MB 00:42 2024-03-25T17:46:20.716 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:46:20.717 INFO:teuthology.orchestra.run.smithi097.stdout:Total 475 kB/s | 20 MB 00:42 2024-03-25T17:46:20.717 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2024-03-25T17:46:20.721 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2024-03-25T17:46:20.721 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2024-03-25T17:46:20.871 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2024-03-25T17:46:20.871 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2024-03-25T17:46:23.105 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2024-03-25T17:46:23.473 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:46:24.448 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:46:24.716 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:46:24.716 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T17:46:24.717 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2024-03-25T17:46:24.717 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:46:24.717 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T17:46:24.717 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-03-25T17:46:24.932 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install ceph-mds-debuginfo 2024-03-25T17:46:25.496 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 1:21:57 ago on Mon 25 Mar 2024 04:24:28 PM UTC. 2024-03-25T17:46:25.643 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-03-25T17:46:25.644 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-03-25T17:46:25.644 INFO:teuthology.orchestra.run.smithi097.stdout: Package Arch Version Repo Size 2024-03-25T17:46:25.644 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-03-25T17:46:25.644 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2024-03-25T17:46:25.644 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mds-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 48 M 2024-03-25T17:46:25.645 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T17:46:25.645 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2024-03-25T17:46:25.645 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-03-25T17:46:25.645 INFO:teuthology.orchestra.run.smithi097.stdout:Install 1 Package 2024-03-25T17:46:25.645 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T17:46:25.645 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 48 M 2024-03-25T17:46:25.645 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 173 M 2024-03-25T17:46:25.645 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2024-03-25T17:49:08.392 INFO:teuthology.orchestra.run.smithi097.stdout:ceph-mds-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 300 kB/s | 48 MB 02:42 2024-03-25T17:49:08.393 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:49:08.393 INFO:teuthology.orchestra.run.smithi097.stdout:Total 300 kB/s | 48 MB 02:42 2024-03-25T17:49:08.393 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2024-03-25T17:49:08.396 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2024-03-25T17:49:08.396 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2024-03-25T17:49:08.711 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2024-03-25T17:49:08.711 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2024-03-25T17:49:13.454 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2024-03-25T17:49:13.599 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:49:14.934 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:49:15.191 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:49:15.191 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T17:49:15.191 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2024-03-25T17:49:15.191 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:49:15.191 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T17:49:15.191 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-03-25T17:49:15.372 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install ceph-mon-debuginfo 2024-03-25T17:49:15.936 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 1:24:47 ago on Mon 25 Mar 2024 04:24:28 PM UTC. 2024-03-25T17:49:16.085 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-03-25T17:49:16.085 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-03-25T17:49:16.085 INFO:teuthology.orchestra.run.smithi097.stdout: Package Arch Version Repo Size 2024-03-25T17:49:16.085 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-03-25T17:49:16.085 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2024-03-25T17:49:16.085 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mon-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 M 2024-03-25T17:49:16.085 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T17:49:16.086 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2024-03-25T17:49:16.086 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-03-25T17:49:16.086 INFO:teuthology.orchestra.run.smithi097.stdout:Install 1 Package 2024-03-25T17:49:16.086 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T17:49:16.086 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 99 M 2024-03-25T17:49:16.086 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 449 M 2024-03-25T17:49:16.086 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2024-03-25T17:51:08.932 INFO:teuthology.orchestra.run.smithi098.stdout:ceph-osd-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 369 kB/s | 287 MB 13:16 2024-03-25T17:51:08.932 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:51:08.933 INFO:teuthology.orchestra.run.smithi098.stdout:Total 369 kB/s | 287 MB 13:16 2024-03-25T17:51:08.933 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-03-25T17:51:08.936 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-03-25T17:51:08.936 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-03-25T17:51:10.651 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-03-25T17:51:10.652 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-03-25T17:51:39.956 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-03-25T17:51:40.283 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:51:41.518 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:51:41.825 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:51:41.825 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T17:51:41.825 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-03-25T17:51:41.825 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:51:41.825 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T17:51:41.825 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-25T17:51:42.082 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph-fuse-debuginfo 2024-03-25T17:51:42.651 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 1:31:37 ago on Mon 25 Mar 2024 04:20:05 PM UTC. 2024-03-25T17:51:42.800 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-25T17:51:42.800 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T17:51:42.801 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repo Size 2024-03-25T17:51:42.801 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T17:51:42.801 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-03-25T17:51:42.801 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.9 M 2024-03-25T17:51:42.801 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T17:51:42.801 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-03-25T17:51:42.801 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T17:51:42.801 INFO:teuthology.orchestra.run.smithi098.stdout:Install 1 Package 2024-03-25T17:51:42.801 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T17:51:42.801 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 5.9 M 2024-03-25T17:51:42.801 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 28 M 2024-03-25T17:51:42.801 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-03-25T17:52:05.407 INFO:teuthology.orchestra.run.smithi098.stdout:ceph-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x 269 kB/s | 5.9 MB 00:22 2024-03-25T17:52:05.408 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:52:05.408 INFO:teuthology.orchestra.run.smithi098.stdout:Total 269 kB/s | 5.9 MB 00:22 2024-03-25T17:52:05.409 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-03-25T17:52:05.413 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-03-25T17:52:05.413 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-03-25T17:52:05.484 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-03-25T17:52:05.484 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-03-25T17:52:06.215 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-03-25T17:52:06.322 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:52:06.859 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:52:07.147 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:52:07.147 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T17:52:07.147 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-03-25T17:52:07.147 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:52:07.147 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T17:52:07.147 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-25T17:52:07.317 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install librados-devel-debuginfo 2024-03-25T17:52:07.875 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 1:32:02 ago on Mon 25 Mar 2024 04:20:05 PM UTC. 2024-03-25T17:52:08.023 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-25T17:52:08.023 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T17:52:08.023 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repo Size 2024-03-25T17:52:08.023 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T17:52:08.024 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-03-25T17:52:08.024 INFO:teuthology.orchestra.run.smithi098.stdout: librados-devel-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 109 k 2024-03-25T17:52:08.024 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T17:52:08.024 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-03-25T17:52:08.024 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T17:52:08.024 INFO:teuthology.orchestra.run.smithi098.stdout:Install 1 Package 2024-03-25T17:52:08.024 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T17:52:08.025 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 109 k 2024-03-25T17:52:08.025 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 440 k 2024-03-25T17:52:08.025 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-03-25T17:52:08.844 INFO:teuthology.orchestra.run.smithi098.stdout:librados-devel-debuginfo-19.0.0-1439.g5dcb190d. 133 kB/s | 109 kB 00:00 2024-03-25T17:52:08.844 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:52:08.845 INFO:teuthology.orchestra.run.smithi098.stdout:Total 132 kB/s | 109 kB 00:00 2024-03-25T17:52:08.845 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-03-25T17:52:08.848 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-03-25T17:52:08.848 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-03-25T17:52:08.884 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-03-25T17:52:08.884 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-03-25T17:52:09.004 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-03-25T17:52:09.110 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T17:52:09.493 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T17:52:09.770 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T17:52:09.770 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T17:52:09.770 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-03-25T17:52:09.770 INFO:teuthology.orchestra.run.smithi098.stdout: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:52:09.770 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T17:52:09.770 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-25T17:52:09.945 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install libcephfs2-debuginfo 2024-03-25T17:52:10.510 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 1:32:05 ago on Mon 25 Mar 2024 04:20:05 PM UTC. 2024-03-25T17:52:10.658 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-25T17:52:10.659 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T17:52:10.659 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repo Size 2024-03-25T17:52:10.659 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T17:52:10.659 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-03-25T17:52:10.659 INFO:teuthology.orchestra.run.smithi098.stdout: libcephfs2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.4 M 2024-03-25T17:52:10.659 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T17:52:10.659 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-03-25T17:52:10.659 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T17:52:10.659 INFO:teuthology.orchestra.run.smithi098.stdout:Install 1 Package 2024-03-25T17:52:10.660 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T17:52:10.660 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 5.4 M 2024-03-25T17:52:10.660 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 25 M 2024-03-25T17:52:10.660 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-03-25T17:52:26.821 INFO:teuthology.orchestra.run.smithi098.stdout:libcephfs2-debuginfo-19.0.0-1439.g5dcb190d.el9. 339 kB/s | 5.4 MB 00:16 2024-03-25T17:52:26.822 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:52:26.825 INFO:teuthology.orchestra.run.smithi098.stdout:Total 339 kB/s | 5.4 MB 00:16 2024-03-25T17:52:26.825 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-03-25T17:52:26.826 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-03-25T17:52:26.826 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-03-25T17:52:26.894 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-03-25T17:52:26.894 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-03-25T17:52:27.562 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-03-25T17:52:27.662 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T17:52:28.185 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T17:52:28.458 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T17:52:28.458 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T17:52:28.459 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-03-25T17:52:28.459 INFO:teuthology.orchestra.run.smithi098.stdout: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:52:28.459 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T17:52:28.459 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-25T17:52:28.646 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install librados2-debuginfo 2024-03-25T17:52:29.210 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 1:32:24 ago on Mon 25 Mar 2024 04:20:05 PM UTC. 2024-03-25T17:52:29.358 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-25T17:52:29.359 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T17:52:29.359 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repo Size 2024-03-25T17:52:29.359 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T17:52:29.359 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-03-25T17:52:29.359 INFO:teuthology.orchestra.run.smithi098.stdout: librados2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 44 M 2024-03-25T17:52:29.359 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T17:52:29.359 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-03-25T17:52:29.359 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T17:52:29.359 INFO:teuthology.orchestra.run.smithi098.stdout:Install 1 Package 2024-03-25T17:52:29.359 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T17:52:29.359 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 44 M 2024-03-25T17:52:29.359 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 253 M 2024-03-25T17:52:29.360 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-03-25T17:54:13.713 INFO:teuthology.orchestra.run.smithi097.stdout:ceph-mon-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 339 kB/s | 99 MB 04:57 2024-03-25T17:54:13.713 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:54:13.714 INFO:teuthology.orchestra.run.smithi097.stdout:Total 339 kB/s | 99 MB 04:57 2024-03-25T17:54:13.714 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2024-03-25T17:54:13.716 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2024-03-25T17:54:13.717 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2024-03-25T17:54:14.325 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2024-03-25T17:54:14.325 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2024-03-25T17:54:24.579 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2024-03-25T17:54:24.667 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:54:27.544 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:54:27.791 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:54:27.792 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T17:54:27.792 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2024-03-25T17:54:27.792 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:54:27.792 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T17:54:27.792 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-03-25T17:54:27.998 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install ceph-osd-debuginfo 2024-03-25T17:54:28.559 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 1:30:00 ago on Mon 25 Mar 2024 04:24:28 PM UTC. 2024-03-25T17:54:28.707 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-03-25T17:54:28.708 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-03-25T17:54:28.708 INFO:teuthology.orchestra.run.smithi097.stdout: Package Arch Version Repo Size 2024-03-25T17:54:28.708 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-03-25T17:54:28.708 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2024-03-25T17:54:28.708 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-osd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 287 M 2024-03-25T17:54:28.708 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T17:54:28.708 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2024-03-25T17:54:28.708 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-03-25T17:54:28.708 INFO:teuthology.orchestra.run.smithi097.stdout:Install 1 Package 2024-03-25T17:54:28.709 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T17:54:28.709 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 287 M 2024-03-25T17:54:28.709 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 1.2 G 2024-03-25T17:54:28.710 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2024-03-25T17:54:39.177 INFO:teuthology.orchestra.run.smithi098.stdout:librados2-debuginfo-19.0.0-1439.g5dcb190d.el9.x 349 kB/s | 44 MB 02:09 2024-03-25T17:54:39.177 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:54:39.178 INFO:teuthology.orchestra.run.smithi098.stdout:Total 349 kB/s | 44 MB 02:09 2024-03-25T17:54:39.178 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-03-25T17:54:39.182 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-03-25T17:54:39.182 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-03-25T17:54:39.482 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-03-25T17:54:39.482 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-03-25T17:54:44.544 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-03-25T17:54:44.642 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:54:46.576 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:54:46.850 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:54:46.851 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T17:54:46.851 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-03-25T17:54:46.851 INFO:teuthology.orchestra.run.smithi098.stdout: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:54:46.851 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T17:54:46.851 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-25T17:54:47.049 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install librbd1-debuginfo 2024-03-25T17:54:47.612 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 1:34:42 ago on Mon 25 Mar 2024 04:20:05 PM UTC. 2024-03-25T17:54:47.763 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-25T17:54:47.764 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T17:54:47.764 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repo Size 2024-03-25T17:54:47.764 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T17:54:47.765 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-03-25T17:54:47.765 INFO:teuthology.orchestra.run.smithi098.stdout: librbd1-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T17:54:47.765 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T17:54:47.765 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-03-25T17:54:47.765 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T17:54:47.765 INFO:teuthology.orchestra.run.smithi098.stdout:Install 1 Package 2024-03-25T17:54:47.765 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T17:54:47.765 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 52 M 2024-03-25T17:54:47.765 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 308 M 2024-03-25T17:54:47.765 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-03-25T17:56:23.854 INFO:teuthology.orchestra.run.smithi098.stdout:librbd1-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 552 kB/s | 52 MB 01:36 2024-03-25T17:56:23.854 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:56:23.855 INFO:teuthology.orchestra.run.smithi098.stdout:Total 552 kB/s | 52 MB 01:36 2024-03-25T17:56:23.855 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-03-25T17:56:23.858 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-03-25T17:56:23.858 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-03-25T17:56:24.201 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-03-25T17:56:24.201 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-03-25T17:56:30.195 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-03-25T17:56:30.412 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:56:32.640 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:56:32.910 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:56:32.910 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T17:56:32.910 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-03-25T17:56:32.910 INFO:teuthology.orchestra.run.smithi098.stdout: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:56:32.910 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T17:56:32.910 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-25T17:56:33.134 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install python3-cephfs-debuginfo 2024-03-25T17:56:33.688 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 1:36:28 ago on Mon 25 Mar 2024 04:20:05 PM UTC. 2024-03-25T17:56:33.836 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-25T17:56:33.837 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T17:56:33.837 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repo Size 2024-03-25T17:56:33.837 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T17:56:33.837 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-03-25T17:56:33.837 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cephfs-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 615 k 2024-03-25T17:56:33.837 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T17:56:33.837 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-03-25T17:56:33.837 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T17:56:33.837 INFO:teuthology.orchestra.run.smithi098.stdout:Install 1 Package 2024-03-25T17:56:33.838 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T17:56:33.838 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 615 k 2024-03-25T17:56:33.838 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 2.1 M 2024-03-25T17:56:33.838 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-03-25T17:56:37.115 INFO:teuthology.orchestra.run.smithi098.stdout:python3-cephfs-debuginfo-19.0.0-1439.g5dcb190d. 188 kB/s | 615 kB 00:03 2024-03-25T17:56:37.116 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:56:37.116 INFO:teuthology.orchestra.run.smithi098.stdout:Total 188 kB/s | 615 kB 00:03 2024-03-25T17:56:37.116 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-03-25T17:56:37.120 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-03-25T17:56:37.121 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-03-25T17:56:37.162 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-03-25T17:56:37.162 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-03-25T17:56:37.329 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-03-25T17:56:37.431 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T17:56:37.817 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T17:56:38.092 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T17:56:38.092 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T17:56:38.092 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-03-25T17:56:38.092 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:56:38.092 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T17:56:38.092 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-25T17:56:38.282 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install python3-rados-debuginfo 2024-03-25T17:56:38.844 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 1:36:33 ago on Mon 25 Mar 2024 04:20:05 PM UTC. 2024-03-25T17:56:38.993 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-25T17:56:38.994 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T17:56:38.994 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repo Size 2024-03-25T17:56:38.994 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T17:56:38.994 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-03-25T17:56:38.994 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rados-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 771 k 2024-03-25T17:56:38.994 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T17:56:38.994 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-03-25T17:56:38.994 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T17:56:38.995 INFO:teuthology.orchestra.run.smithi098.stdout:Install 1 Package 2024-03-25T17:56:38.995 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T17:56:38.995 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 771 k 2024-03-25T17:56:38.995 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 2.7 M 2024-03-25T17:56:38.995 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-03-25T17:56:41.510 INFO:teuthology.orchestra.run.smithi098.stdout:python3-rados-debuginfo-19.0.0-1439.g5dcb190d.e 307 kB/s | 771 kB 00:02 2024-03-25T17:56:41.511 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:56:41.511 INFO:teuthology.orchestra.run.smithi098.stdout:Total 306 kB/s | 771 kB 00:02 2024-03-25T17:56:41.511 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-03-25T17:56:41.515 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-03-25T17:56:41.515 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-03-25T17:56:41.556 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-03-25T17:56:41.556 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-03-25T17:56:41.735 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-03-25T17:56:41.834 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T17:56:42.206 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T17:56:42.484 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T17:56:42.484 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T17:56:42.484 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-03-25T17:56:42.484 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:56:42.484 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T17:56:42.484 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-25T17:56:42.742 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install python3-rbd-debuginfo 2024-03-25T17:56:43.300 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 1:36:38 ago on Mon 25 Mar 2024 04:20:05 PM UTC. 2024-03-25T17:56:43.448 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-25T17:56:43.449 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T17:56:43.449 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repo Size 2024-03-25T17:56:43.449 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T17:56:43.449 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-03-25T17:56:43.449 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.1 M 2024-03-25T17:56:43.449 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T17:56:43.449 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-03-25T17:56:43.449 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T17:56:43.449 INFO:teuthology.orchestra.run.smithi098.stdout:Install 1 Package 2024-03-25T17:56:43.449 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T17:56:43.450 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 1.1 M 2024-03-25T17:56:43.450 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 4.3 M 2024-03-25T17:56:43.450 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-03-25T17:56:48.621 INFO:teuthology.orchestra.run.smithi098.stdout:python3-rbd-debuginfo-19.0.0-1439.g5dcb190d.el9 223 kB/s | 1.1 MB 00:05 2024-03-25T17:56:48.622 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:56:48.622 INFO:teuthology.orchestra.run.smithi098.stdout:Total 222 kB/s | 1.1 MB 00:05 2024-03-25T17:56:48.623 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-03-25T17:56:48.627 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-03-25T17:56:48.627 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-03-25T17:56:48.670 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-03-25T17:56:48.671 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-03-25T17:56:48.888 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-03-25T17:56:48.995 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:56:49.399 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:56:49.675 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:56:49.675 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T17:56:49.675 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-03-25T17:56:49.676 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:56:49.676 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T17:56:49.676 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-25T17:56:49.905 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install python3-rgw-debuginfo 2024-03-25T17:56:50.471 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 1:36:45 ago on Mon 25 Mar 2024 04:20:05 PM UTC. 2024-03-25T17:56:50.622 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-25T17:56:50.622 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T17:56:50.623 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repo Size 2024-03-25T17:56:50.623 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T17:56:50.623 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-03-25T17:56:50.623 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 337 k 2024-03-25T17:56:50.623 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T17:56:50.623 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-03-25T17:56:50.623 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T17:56:50.623 INFO:teuthology.orchestra.run.smithi098.stdout:Install 1 Package 2024-03-25T17:56:50.623 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T17:56:50.623 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 337 k 2024-03-25T17:56:50.623 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 1.0 M 2024-03-25T17:56:50.623 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-03-25T17:56:53.033 INFO:teuthology.orchestra.run.smithi098.stdout:python3-rgw-debuginfo-19.0.0-1439.g5dcb190d.el9 140 kB/s | 337 kB 00:02 2024-03-25T17:56:53.033 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:56:53.034 INFO:teuthology.orchestra.run.smithi098.stdout:Total 140 kB/s | 337 kB 00:02 2024-03-25T17:56:53.034 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-03-25T17:56:53.038 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-03-25T17:56:53.038 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-03-25T17:56:53.077 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-03-25T17:56:53.077 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-03-25T17:56:53.216 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-03-25T17:56:53.323 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:56:53.679 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:56:53.940 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:56:53.940 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T17:56:53.940 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-03-25T17:56:53.940 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:56:53.940 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T17:56:53.940 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-25T17:56:54.158 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install rbd-fuse-debuginfo 2024-03-25T17:56:54.712 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 1:36:49 ago on Mon 25 Mar 2024 04:20:05 PM UTC. 2024-03-25T17:56:54.861 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-25T17:56:54.862 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T17:56:54.862 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repo Size 2024-03-25T17:56:54.862 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T17:56:54.862 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-03-25T17:56:54.862 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 509 k 2024-03-25T17:56:54.863 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T17:56:54.863 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-03-25T17:56:54.863 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T17:56:54.863 INFO:teuthology.orchestra.run.smithi098.stdout:Install 1 Package 2024-03-25T17:56:54.863 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T17:56:54.863 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 509 k 2024-03-25T17:56:54.863 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 2.5 M 2024-03-25T17:56:54.863 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-03-25T17:56:58.419 INFO:teuthology.orchestra.run.smithi098.stdout:rbd-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 143 kB/s | 509 kB 00:03 2024-03-25T17:56:58.420 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:56:58.420 INFO:teuthology.orchestra.run.smithi098.stdout:Total 143 kB/s | 509 kB 00:03 2024-03-25T17:56:58.421 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-03-25T17:56:58.425 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-03-25T17:56:58.425 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-03-25T17:56:58.463 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-03-25T17:56:58.464 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-03-25T17:56:58.619 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-03-25T17:56:58.717 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:56:59.143 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:56:59.446 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:56:59.447 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T17:56:59.447 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-03-25T17:56:59.447 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:56:59.447 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T17:56:59.447 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-25T17:56:59.652 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install rbd-mirror-debuginfo 2024-03-25T17:57:00.211 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 1:36:55 ago on Mon 25 Mar 2024 04:20:05 PM UTC. 2024-03-25T17:57:00.364 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-25T17:57:00.364 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T17:57:00.365 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repo Size 2024-03-25T17:57:00.365 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T17:57:00.365 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-03-25T17:57:00.365 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-mirror-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T17:57:00.365 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T17:57:00.365 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-03-25T17:57:00.365 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T17:57:00.365 INFO:teuthology.orchestra.run.smithi098.stdout:Install 1 Package 2024-03-25T17:57:00.365 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T17:57:00.366 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 52 M 2024-03-25T17:57:00.366 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 298 M 2024-03-25T17:57:00.366 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-03-25T17:58:42.446 INFO:teuthology.orchestra.run.smithi098.stdout:rbd-mirror-debuginfo-19.0.0-1439.g5dcb190d.el9. 517 kB/s | 52 MB 01:42 2024-03-25T17:58:42.447 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:58:42.447 INFO:teuthology.orchestra.run.smithi098.stdout:Total 517 kB/s | 52 MB 01:42 2024-03-25T17:58:42.447 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-03-25T17:58:42.450 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-03-25T17:58:42.450 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-03-25T17:58:42.783 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-03-25T17:58:42.783 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-03-25T17:58:48.675 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-03-25T17:58:48.760 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T17:58:50.880 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T17:58:51.157 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T17:58:51.158 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T17:58:51.158 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-03-25T17:58:51.158 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:58:51.158 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T17:58:51.158 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-25T17:58:51.365 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install rbd-nbd-debuginfo 2024-03-25T17:58:51.930 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 1:38:46 ago on Mon 25 Mar 2024 04:20:05 PM UTC. 2024-03-25T17:58:52.079 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-25T17:58:52.080 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T17:58:52.080 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repo Size 2024-03-25T17:58:52.080 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T17:58:52.080 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-03-25T17:58:52.080 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-nbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.3 M 2024-03-25T17:58:52.080 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T17:58:52.081 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-03-25T17:58:52.081 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T17:58:52.081 INFO:teuthology.orchestra.run.smithi098.stdout:Install 1 Package 2024-03-25T17:58:52.081 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T17:58:52.081 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 1.3 M 2024-03-25T17:58:52.081 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 5.5 M 2024-03-25T17:58:52.081 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-03-25T17:58:59.322 INFO:teuthology.orchestra.run.smithi098.stdout:rbd-nbd-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 177 kB/s | 1.3 MB 00:07 2024-03-25T17:58:59.323 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:58:59.323 INFO:teuthology.orchestra.run.smithi098.stdout:Total 177 kB/s | 1.3 MB 00:07 2024-03-25T17:58:59.324 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-03-25T17:58:59.328 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-03-25T17:58:59.328 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-03-25T17:58:59.372 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-03-25T17:58:59.372 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-03-25T17:58:59.634 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-03-25T17:58:59.805 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:59:00.163 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:59:00.432 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:59:00.432 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T17:59:00.432 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-03-25T17:59:00.432 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:59:00.432 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T17:59:00.432 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-25T17:59:00.625 DEBUG:teuthology.parallel:result is None 2024-03-25T18:06:16.229 INFO:teuthology.orchestra.run.smithi097.stdout:ceph-osd-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 415 kB/s | 287 MB 11:47 2024-03-25T18:06:16.230 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:06:16.230 INFO:teuthology.orchestra.run.smithi097.stdout:Total 415 kB/s | 287 MB 11:47 2024-03-25T18:06:16.230 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2024-03-25T18:06:16.233 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2024-03-25T18:06:16.233 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2024-03-25T18:06:17.946 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2024-03-25T18:06:17.946 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2024-03-25T18:06:47.351 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2024-03-25T18:06:47.451 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:06:48.645 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:06:48.914 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:06:48.914 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T18:06:48.914 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2024-03-25T18:06:48.914 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:06:48.915 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T18:06:48.915 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-03-25T18:06:49.145 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install ceph-fuse-debuginfo 2024-03-25T18:06:49.710 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 1:42:21 ago on Mon 25 Mar 2024 04:24:28 PM UTC. 2024-03-25T18:06:49.858 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-03-25T18:06:49.858 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-03-25T18:06:49.858 INFO:teuthology.orchestra.run.smithi097.stdout: Package Arch Version Repo Size 2024-03-25T18:06:49.859 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-03-25T18:06:49.859 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2024-03-25T18:06:49.859 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.9 M 2024-03-25T18:06:49.859 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T18:06:49.859 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2024-03-25T18:06:49.859 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-03-25T18:06:49.859 INFO:teuthology.orchestra.run.smithi097.stdout:Install 1 Package 2024-03-25T18:06:49.859 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T18:06:49.859 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 5.9 M 2024-03-25T18:06:49.859 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 28 M 2024-03-25T18:06:49.859 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2024-03-25T18:07:11.521 INFO:teuthology.orchestra.run.smithi097.stdout:ceph-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x 281 kB/s | 5.9 MB 00:21 2024-03-25T18:07:11.522 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:07:11.522 INFO:teuthology.orchestra.run.smithi097.stdout:Total 281 kB/s | 5.9 MB 00:21 2024-03-25T18:07:11.522 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2024-03-25T18:07:11.526 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2024-03-25T18:07:11.526 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2024-03-25T18:07:11.597 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2024-03-25T18:07:11.597 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2024-03-25T18:07:12.326 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2024-03-25T18:07:12.427 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:07:12.996 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:07:13.246 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:07:13.246 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T18:07:13.246 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2024-03-25T18:07:13.246 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:07:13.246 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T18:07:13.247 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-03-25T18:07:13.451 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install librados-devel-debuginfo 2024-03-25T18:07:14.011 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 1:42:46 ago on Mon 25 Mar 2024 04:24:28 PM UTC. 2024-03-25T18:07:14.160 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-03-25T18:07:14.160 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-03-25T18:07:14.160 INFO:teuthology.orchestra.run.smithi097.stdout: Package Arch Version Repo Size 2024-03-25T18:07:14.160 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-03-25T18:07:14.161 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2024-03-25T18:07:14.161 INFO:teuthology.orchestra.run.smithi097.stdout: librados-devel-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 109 k 2024-03-25T18:07:14.161 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T18:07:14.161 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2024-03-25T18:07:14.161 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-03-25T18:07:14.161 INFO:teuthology.orchestra.run.smithi097.stdout:Install 1 Package 2024-03-25T18:07:14.161 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T18:07:14.161 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 109 k 2024-03-25T18:07:14.161 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 440 k 2024-03-25T18:07:14.161 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2024-03-25T18:07:14.857 INFO:teuthology.orchestra.run.smithi097.stdout:librados-devel-debuginfo-19.0.0-1439.g5dcb190d. 156 kB/s | 109 kB 00:00 2024-03-25T18:07:14.857 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:07:14.858 INFO:teuthology.orchestra.run.smithi097.stdout:Total 156 kB/s | 109 kB 00:00 2024-03-25T18:07:14.858 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2024-03-25T18:07:14.861 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2024-03-25T18:07:14.861 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2024-03-25T18:07:14.897 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2024-03-25T18:07:14.898 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2024-03-25T18:07:15.016 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2024-03-25T18:07:15.122 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:07:15.514 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:07:15.771 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:07:15.771 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T18:07:15.772 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2024-03-25T18:07:15.772 INFO:teuthology.orchestra.run.smithi097.stdout: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:07:15.772 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T18:07:15.772 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-03-25T18:07:15.972 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install libcephfs2-debuginfo 2024-03-25T18:07:16.535 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 1:42:48 ago on Mon 25 Mar 2024 04:24:28 PM UTC. 2024-03-25T18:07:16.684 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-03-25T18:07:16.685 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-03-25T18:07:16.685 INFO:teuthology.orchestra.run.smithi097.stdout: Package Arch Version Repo Size 2024-03-25T18:07:16.685 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-03-25T18:07:16.685 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2024-03-25T18:07:16.685 INFO:teuthology.orchestra.run.smithi097.stdout: libcephfs2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.4 M 2024-03-25T18:07:16.685 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T18:07:16.685 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2024-03-25T18:07:16.686 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-03-25T18:07:16.686 INFO:teuthology.orchestra.run.smithi097.stdout:Install 1 Package 2024-03-25T18:07:16.686 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T18:07:16.686 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 5.4 M 2024-03-25T18:07:16.686 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 25 M 2024-03-25T18:07:16.686 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2024-03-25T18:07:37.583 INFO:teuthology.orchestra.run.smithi097.stdout:libcephfs2-debuginfo-19.0.0-1439.g5dcb190d.el9. 262 kB/s | 5.4 MB 00:20 2024-03-25T18:07:37.584 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:07:37.584 INFO:teuthology.orchestra.run.smithi097.stdout:Total 262 kB/s | 5.4 MB 00:20 2024-03-25T18:07:37.585 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2024-03-25T18:07:37.588 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2024-03-25T18:07:37.589 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2024-03-25T18:07:37.658 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2024-03-25T18:07:37.658 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2024-03-25T18:07:38.326 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2024-03-25T18:07:38.432 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:07:38.992 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:07:39.244 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:07:39.244 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T18:07:39.244 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2024-03-25T18:07:39.244 INFO:teuthology.orchestra.run.smithi097.stdout: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:07:39.245 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T18:07:39.245 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-03-25T18:07:39.439 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install librados2-debuginfo 2024-03-25T18:07:40.000 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 1:43:11 ago on Mon 25 Mar 2024 04:24:28 PM UTC. 2024-03-25T18:07:40.147 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-03-25T18:07:40.148 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-03-25T18:07:40.148 INFO:teuthology.orchestra.run.smithi097.stdout: Package Arch Version Repo Size 2024-03-25T18:07:40.148 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-03-25T18:07:40.148 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2024-03-25T18:07:40.148 INFO:teuthology.orchestra.run.smithi097.stdout: librados2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 44 M 2024-03-25T18:07:40.148 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T18:07:40.148 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2024-03-25T18:07:40.149 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-03-25T18:07:40.149 INFO:teuthology.orchestra.run.smithi097.stdout:Install 1 Package 2024-03-25T18:07:40.149 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T18:07:40.149 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 44 M 2024-03-25T18:07:40.149 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 253 M 2024-03-25T18:07:40.149 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2024-03-25T18:09:53.539 INFO:teuthology.orchestra.run.smithi097.stdout:librados2-debuginfo-19.0.0-1439.g5dcb190d.el9.x 340 kB/s | 44 MB 02:13 2024-03-25T18:09:53.540 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:09:53.540 INFO:teuthology.orchestra.run.smithi097.stdout:Total 340 kB/s | 44 MB 02:13 2024-03-25T18:09:53.540 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2024-03-25T18:09:53.543 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2024-03-25T18:09:53.543 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2024-03-25T18:09:53.844 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2024-03-25T18:09:53.844 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2024-03-25T18:09:58.872 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2024-03-25T18:09:58.970 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:10:00.805 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:10:01.050 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:10:01.051 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T18:10:01.051 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2024-03-25T18:10:01.051 INFO:teuthology.orchestra.run.smithi097.stdout: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:10:01.051 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T18:10:01.051 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-03-25T18:10:01.270 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install librbd1-debuginfo 2024-03-25T18:10:01.852 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 1:45:33 ago on Mon 25 Mar 2024 04:24:28 PM UTC. 2024-03-25T18:10:02.008 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-03-25T18:10:02.008 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-03-25T18:10:02.009 INFO:teuthology.orchestra.run.smithi097.stdout: Package Arch Version Repo Size 2024-03-25T18:10:02.009 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-03-25T18:10:02.009 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2024-03-25T18:10:02.009 INFO:teuthology.orchestra.run.smithi097.stdout: librbd1-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T18:10:02.009 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T18:10:02.009 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2024-03-25T18:10:02.009 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-03-25T18:10:02.009 INFO:teuthology.orchestra.run.smithi097.stdout:Install 1 Package 2024-03-25T18:10:02.009 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T18:10:02.009 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 52 M 2024-03-25T18:10:02.009 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 308 M 2024-03-25T18:10:02.009 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2024-03-25T18:11:37.073 INFO:teuthology.orchestra.run.smithi097.stdout:librbd1-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 558 kB/s | 52 MB 01:35 2024-03-25T18:11:37.074 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:11:37.074 INFO:teuthology.orchestra.run.smithi097.stdout:Total 558 kB/s | 52 MB 01:35 2024-03-25T18:11:37.074 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2024-03-25T18:11:37.078 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2024-03-25T18:11:37.078 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2024-03-25T18:11:37.422 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2024-03-25T18:11:37.422 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2024-03-25T18:11:43.438 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2024-03-25T18:11:43.629 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:11:45.826 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:11:46.085 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:11:46.086 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T18:11:46.086 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2024-03-25T18:11:46.086 INFO:teuthology.orchestra.run.smithi097.stdout: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:11:46.086 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T18:11:46.086 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-03-25T18:11:46.300 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install python3-cephfs-debuginfo 2024-03-25T18:11:46.863 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 1:47:18 ago on Mon 25 Mar 2024 04:24:28 PM UTC. 2024-03-25T18:11:47.011 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-03-25T18:11:47.012 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-03-25T18:11:47.012 INFO:teuthology.orchestra.run.smithi097.stdout: Package Arch Version Repo Size 2024-03-25T18:11:47.012 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-03-25T18:11:47.012 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2024-03-25T18:11:47.012 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cephfs-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 615 k 2024-03-25T18:11:47.012 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T18:11:47.012 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2024-03-25T18:11:47.012 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-03-25T18:11:47.012 INFO:teuthology.orchestra.run.smithi097.stdout:Install 1 Package 2024-03-25T18:11:47.012 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T18:11:47.012 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 615 k 2024-03-25T18:11:47.012 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 2.1 M 2024-03-25T18:11:47.013 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2024-03-25T18:11:48.584 INFO:teuthology.orchestra.run.smithi097.stdout:python3-cephfs-debuginfo-19.0.0-1439.g5dcb190d. 391 kB/s | 615 kB 00:01 2024-03-25T18:11:48.584 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:11:48.585 INFO:teuthology.orchestra.run.smithi097.stdout:Total 391 kB/s | 615 kB 00:01 2024-03-25T18:11:48.585 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2024-03-25T18:11:48.589 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2024-03-25T18:11:48.589 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2024-03-25T18:11:48.631 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2024-03-25T18:11:48.632 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2024-03-25T18:11:48.823 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2024-03-25T18:11:48.925 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:11:49.341 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:11:49.593 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:11:49.593 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T18:11:49.593 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2024-03-25T18:11:49.594 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:11:49.594 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T18:11:49.594 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-03-25T18:11:49.797 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install python3-rados-debuginfo 2024-03-25T18:11:50.362 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 1:47:22 ago on Mon 25 Mar 2024 04:24:28 PM UTC. 2024-03-25T18:11:50.512 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-03-25T18:11:50.513 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-03-25T18:11:50.513 INFO:teuthology.orchestra.run.smithi097.stdout: Package Arch Version Repo Size 2024-03-25T18:11:50.513 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-03-25T18:11:50.513 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2024-03-25T18:11:50.513 INFO:teuthology.orchestra.run.smithi097.stdout: python3-rados-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 771 k 2024-03-25T18:11:50.513 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T18:11:50.513 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2024-03-25T18:11:50.514 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-03-25T18:11:50.514 INFO:teuthology.orchestra.run.smithi097.stdout:Install 1 Package 2024-03-25T18:11:50.514 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T18:11:50.514 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 771 k 2024-03-25T18:11:50.514 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 2.7 M 2024-03-25T18:11:50.514 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2024-03-25T18:11:52.062 INFO:teuthology.orchestra.run.smithi097.stdout:python3-rados-debuginfo-19.0.0-1439.g5dcb190d.e 498 kB/s | 771 kB 00:01 2024-03-25T18:11:52.063 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:11:52.063 INFO:teuthology.orchestra.run.smithi097.stdout:Total 498 kB/s | 771 kB 00:01 2024-03-25T18:11:52.063 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2024-03-25T18:11:52.067 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2024-03-25T18:11:52.067 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2024-03-25T18:11:52.108 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2024-03-25T18:11:52.109 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2024-03-25T18:11:52.294 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2024-03-25T18:11:52.402 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T18:11:52.802 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T18:11:53.060 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T18:11:53.060 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T18:11:53.060 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2024-03-25T18:11:53.060 INFO:teuthology.orchestra.run.smithi097.stdout: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:11:53.060 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T18:11:53.060 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-03-25T18:11:53.285 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install python3-rbd-debuginfo 2024-03-25T18:11:53.851 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 1:47:25 ago on Mon 25 Mar 2024 04:24:28 PM UTC. 2024-03-25T18:11:53.999 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-03-25T18:11:54.000 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-03-25T18:11:54.000 INFO:teuthology.orchestra.run.smithi097.stdout: Package Arch Version Repo Size 2024-03-25T18:11:54.000 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-03-25T18:11:54.000 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2024-03-25T18:11:54.000 INFO:teuthology.orchestra.run.smithi097.stdout: python3-rbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.1 M 2024-03-25T18:11:54.000 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T18:11:54.001 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2024-03-25T18:11:54.001 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-03-25T18:11:54.001 INFO:teuthology.orchestra.run.smithi097.stdout:Install 1 Package 2024-03-25T18:11:54.001 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T18:11:54.001 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 1.1 M 2024-03-25T18:11:54.001 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 4.3 M 2024-03-25T18:11:54.001 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2024-03-25T18:11:58.840 INFO:teuthology.orchestra.run.smithi097.stdout:python3-rbd-debuginfo-19.0.0-1439.g5dcb190d.el9 238 kB/s | 1.1 MB 00:04 2024-03-25T18:11:58.840 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:11:58.841 INFO:teuthology.orchestra.run.smithi097.stdout:Total 238 kB/s | 1.1 MB 00:04 2024-03-25T18:11:58.841 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2024-03-25T18:11:58.845 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2024-03-25T18:11:58.845 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2024-03-25T18:11:58.886 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2024-03-25T18:11:58.887 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2024-03-25T18:11:59.106 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2024-03-25T18:11:59.207 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:11:59.619 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:11:59.868 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:11:59.868 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T18:11:59.868 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2024-03-25T18:11:59.868 INFO:teuthology.orchestra.run.smithi097.stdout: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:11:59.868 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T18:11:59.868 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-03-25T18:12:00.087 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install python3-rgw-debuginfo 2024-03-25T18:12:00.648 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 1:47:32 ago on Mon 25 Mar 2024 04:24:28 PM UTC. 2024-03-25T18:12:00.797 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-03-25T18:12:00.798 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-03-25T18:12:00.798 INFO:teuthology.orchestra.run.smithi097.stdout: Package Arch Version Repo Size 2024-03-25T18:12:00.798 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-03-25T18:12:00.798 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2024-03-25T18:12:00.798 INFO:teuthology.orchestra.run.smithi097.stdout: python3-rgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 337 k 2024-03-25T18:12:00.798 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T18:12:00.798 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2024-03-25T18:12:00.798 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-03-25T18:12:00.798 INFO:teuthology.orchestra.run.smithi097.stdout:Install 1 Package 2024-03-25T18:12:00.798 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T18:12:00.799 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 337 k 2024-03-25T18:12:00.799 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 1.0 M 2024-03-25T18:12:00.799 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2024-03-25T18:12:02.176 INFO:teuthology.orchestra.run.smithi097.stdout:python3-rgw-debuginfo-19.0.0-1439.g5dcb190d.el9 245 kB/s | 337 kB 00:01 2024-03-25T18:12:02.176 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:12:02.177 INFO:teuthology.orchestra.run.smithi097.stdout:Total 245 kB/s | 337 kB 00:01 2024-03-25T18:12:02.177 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2024-03-25T18:12:02.181 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2024-03-25T18:12:02.181 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2024-03-25T18:12:02.221 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2024-03-25T18:12:02.222 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2024-03-25T18:12:02.360 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2024-03-25T18:12:02.476 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:12:02.845 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:12:03.110 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:12:03.110 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T18:12:03.110 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2024-03-25T18:12:03.110 INFO:teuthology.orchestra.run.smithi097.stdout: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:12:03.110 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T18:12:03.110 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-03-25T18:12:03.318 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install rbd-fuse-debuginfo 2024-03-25T18:12:03.880 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 1:47:35 ago on Mon 25 Mar 2024 04:24:28 PM UTC. 2024-03-25T18:12:04.028 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-03-25T18:12:04.028 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-03-25T18:12:04.028 INFO:teuthology.orchestra.run.smithi097.stdout: Package Arch Version Repo Size 2024-03-25T18:12:04.029 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-03-25T18:12:04.029 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2024-03-25T18:12:04.029 INFO:teuthology.orchestra.run.smithi097.stdout: rbd-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 509 k 2024-03-25T18:12:04.029 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T18:12:04.029 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2024-03-25T18:12:04.029 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-03-25T18:12:04.029 INFO:teuthology.orchestra.run.smithi097.stdout:Install 1 Package 2024-03-25T18:12:04.029 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T18:12:04.029 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 509 k 2024-03-25T18:12:04.030 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 2.5 M 2024-03-25T18:12:04.030 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2024-03-25T18:12:06.412 INFO:teuthology.orchestra.run.smithi097.stdout:rbd-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 214 kB/s | 509 kB 00:02 2024-03-25T18:12:06.413 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:12:06.413 INFO:teuthology.orchestra.run.smithi097.stdout:Total 214 kB/s | 509 kB 00:02 2024-03-25T18:12:06.413 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2024-03-25T18:12:06.417 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2024-03-25T18:12:06.417 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2024-03-25T18:12:06.457 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2024-03-25T18:12:06.458 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2024-03-25T18:12:06.624 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2024-03-25T18:12:06.729 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:12:07.087 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:12:07.341 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:12:07.341 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T18:12:07.341 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2024-03-25T18:12:07.341 INFO:teuthology.orchestra.run.smithi097.stdout: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:12:07.341 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T18:12:07.342 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-03-25T18:12:07.551 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install rbd-mirror-debuginfo 2024-03-25T18:12:08.113 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 1:47:40 ago on Mon 25 Mar 2024 04:24:28 PM UTC. 2024-03-25T18:12:08.264 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-03-25T18:12:08.265 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-03-25T18:12:08.265 INFO:teuthology.orchestra.run.smithi097.stdout: Package Arch Version Repo Size 2024-03-25T18:12:08.265 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-03-25T18:12:08.265 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2024-03-25T18:12:08.265 INFO:teuthology.orchestra.run.smithi097.stdout: rbd-mirror-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T18:12:08.265 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T18:12:08.265 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2024-03-25T18:12:08.265 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-03-25T18:12:08.265 INFO:teuthology.orchestra.run.smithi097.stdout:Install 1 Package 2024-03-25T18:12:08.265 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T18:12:08.266 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 52 M 2024-03-25T18:12:08.266 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 298 M 2024-03-25T18:12:08.266 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2024-03-25T18:15:07.104 INFO:teuthology.orchestra.run.smithi097.stdout:rbd-mirror-debuginfo-19.0.0-1439.g5dcb190d.el9. 295 kB/s | 52 MB 02:58 2024-03-25T18:15:07.105 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:15:07.105 INFO:teuthology.orchestra.run.smithi097.stdout:Total 295 kB/s | 52 MB 02:58 2024-03-25T18:15:07.105 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2024-03-25T18:15:07.108 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2024-03-25T18:15:07.108 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2024-03-25T18:15:07.442 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2024-03-25T18:15:07.442 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2024-03-25T18:15:13.327 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2024-03-25T18:15:13.527 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:15:15.630 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:15:15.891 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:15:15.891 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T18:15:15.891 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2024-03-25T18:15:15.891 INFO:teuthology.orchestra.run.smithi097.stdout: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:15:15.891 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T18:15:15.891 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-03-25T18:15:16.100 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install rbd-nbd-debuginfo 2024-03-25T18:15:16.666 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 1:50:48 ago on Mon 25 Mar 2024 04:24:28 PM UTC. 2024-03-25T18:15:16.817 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-03-25T18:15:16.817 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-03-25T18:15:16.817 INFO:teuthology.orchestra.run.smithi097.stdout: Package Arch Version Repo Size 2024-03-25T18:15:16.817 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-03-25T18:15:16.817 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2024-03-25T18:15:16.818 INFO:teuthology.orchestra.run.smithi097.stdout: rbd-nbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.3 M 2024-03-25T18:15:16.818 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T18:15:16.818 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2024-03-25T18:15:16.818 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-03-25T18:15:16.818 INFO:teuthology.orchestra.run.smithi097.stdout:Install 1 Package 2024-03-25T18:15:16.818 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T18:15:16.818 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 1.3 M 2024-03-25T18:15:16.819 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 5.5 M 2024-03-25T18:15:16.819 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2024-03-25T18:15:20.589 INFO:teuthology.orchestra.run.smithi097.stdout:rbd-nbd-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 340 kB/s | 1.3 MB 00:03 2024-03-25T18:15:20.590 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:15:20.590 INFO:teuthology.orchestra.run.smithi097.stdout:Total 340 kB/s | 1.3 MB 00:03 2024-03-25T18:15:20.591 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2024-03-25T18:15:20.595 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2024-03-25T18:15:20.595 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2024-03-25T18:15:20.639 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2024-03-25T18:15:20.639 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2024-03-25T18:15:20.906 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2024-03-25T18:15:20.997 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:15:21.389 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:15:21.649 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:15:21.649 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T18:15:21.649 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2024-03-25T18:15:21.649 INFO:teuthology.orchestra.run.smithi097.stdout: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:15:21.649 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T18:15:21.650 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-03-25T18:15:21.832 DEBUG:teuthology.parallel:result is None 2024-03-25T18:15:21.832 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T18:15:22.053 DEBUG:teuthology.orchestra.run.smithi097:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-25T18:15:22.074 INFO:teuthology.orchestra.run.smithi097.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-25T18:15:22.074 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-25T18:15:22.074 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-25T18:15:22.076 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T18:15:22.218 DEBUG:teuthology.orchestra.run.smithi098:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-25T18:15:22.243 INFO:teuthology.orchestra.run.smithi098.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-25T18:15:22.244 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-25T18:15:22.244 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-25T18:15:22.246 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-25T18:15:22.246 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-25T18:15:22.246 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-25T18:15:22.273 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-25T18:15:22.273 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-25T18:15:22.314 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-25T18:15:22.314 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-25T18:15:22.314 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd of=/usr/bin/daemon-helper 2024-03-25T18:15:22.341 DEBUG:teuthology.orchestra.run.smithi097:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-25T18:15:22.409 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-25T18:15:22.409 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/usr/bin/daemon-helper 2024-03-25T18:15:22.438 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-25T18:15:22.507 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-25T18:15:22.508 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-25T18:15:22.508 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-25T18:15:22.534 DEBUG:teuthology.orchestra.run.smithi097:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-25T18:15:22.601 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-25T18:15:22.601 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-25T18:15:22.630 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-25T18:15:22.697 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-25T18:15:22.697 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-25T18:15:22.698 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd of=/usr/bin/stdin-killer 2024-03-25T18:15:22.724 DEBUG:teuthology.orchestra.run.smithi097:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-25T18:15:22.791 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-25T18:15:22.792 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/usr/bin/stdin-killer 2024-03-25T18:15:22.819 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-25T18:15:22.887 INFO:teuthology.run_tasks:Running task ceph... 2024-03-25T18:15:22.962 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-25T18:15:22.962 DEBUG:teuthology.orchestra.run.smithi097:> sudo chmod 777 /var/log/ceph 2024-03-25T18:15:22.965 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 777 /var/log/ceph 2024-03-25T18:15:22.992 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-25T18:15:22.992 DEBUG:teuthology.orchestra.run.smithi097:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-25T18:15:23.031 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-25T18:15:23.060 INFO:tasks.ceph:Creating extra log directories... 2024-03-25T18:15:23.060 DEBUG:teuthology.orchestra.run.smithi097:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-25T18:15:23.098 DEBUG:teuthology.orchestra.run.smithi098:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-25T18:15:23.153 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-25T18:15:23.153 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': False, 'mon_bind_addrvec': True} 2024-03-25T18:15:23.153 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621258', 'branch': 'main', 'description': 'rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind}', 'email': None, 'first_in_suite': False, 'job_id': '7621258', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'mon_bind_msgr2': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'valgrind': {'mds': ['--tool=memcheck'], 'mon': ['--tool=memcheck', '--leak-check=full', '--show-reachable=yes'], 'osd': ['--tool=memcheck']}}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'main', 'sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324'}}, 'owner': 'scheduled_amathuri@teuthology', 'priority': 150, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 6785, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'sleep_before_teardown': 0, 'subset': '99/1000', 'suite': 'rados:verify', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324', 'targets': {'smithi097.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAlUFkA3EhV5kb2ygNpF5qBDAejAgsqx9lIlhqLrnXLtG4ypJAUAbWNTJwBXqf2oFG5BOn1WPktEm7I8kPXy5/4=', 'smithi098.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBG5PMryiLKOLykE/21YnL5gEtLKYPstczuT3fgQ4kfaLo8aFaCr4KeWyCXgjdzw37FS+H3mKEFtX3vKa3yCpmuU='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'thrashosds': {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200}}, {'workunit': {'clients': {'client.0': ['rados/test.sh']}, 'env': {'ALLOW_TIMEOUTS': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-25_13:28:33', 'tube': 'smithi', 'user': 'amathuri', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-25T18:15:23.153 DEBUG:teuthology.orchestra.run.smithi097:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-25T18:15:23.165 DEBUG:teuthology.orchestra.run.smithi098:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-25T18:15:23.183 DEBUG:teuthology.orchestra.run.smithi097:> sudo install -d -m0777 -- /var/run/ceph 2024-03-25T18:15:23.220 DEBUG:teuthology.orchestra.run.smithi098:> sudo install -d -m0777 -- /var/run/ceph 2024-03-25T18:15:23.249 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-25T18:15:23.249 DEBUG:teuthology.orchestra.run.smithi097:> dd if=/scratch_devs of=/dev/stdout 2024-03-25T18:15:23.300 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-25T18:15:23.300 DEBUG:teuthology.orchestra.run.smithi097:> stat /dev/vg_nvme/lv_1 2024-03-25T18:15:23.355 INFO:teuthology.orchestra.run.smithi097.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-25T18:15:23.355 INFO:teuthology.orchestra.run.smithi097.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:15:23.355 INFO:teuthology.orchestra.run.smithi097.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-03-25T18:15:23.355 INFO:teuthology.orchestra.run.smithi097.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:15:23.355 INFO:teuthology.orchestra.run.smithi097.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:15:23.355 INFO:teuthology.orchestra.run.smithi097.stdout:Access: 2024-03-25 16:29:15.638559847 +0000 2024-03-25T18:15:23.355 INFO:teuthology.orchestra.run.smithi097.stdout:Modify: 2024-03-25 16:19:17.835087139 +0000 2024-03-25T18:15:23.355 INFO:teuthology.orchestra.run.smithi097.stdout:Change: 2024-03-25 16:19:17.835087139 +0000 2024-03-25T18:15:23.356 INFO:teuthology.orchestra.run.smithi097.stdout: Birth: 2024-03-25 16:19:17.835087139 +0000 2024-03-25T18:15:23.356 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-25T18:15:23.420 INFO:teuthology.orchestra.run.smithi097.stderr:1+0 records in 2024-03-25T18:15:23.420 INFO:teuthology.orchestra.run.smithi097.stderr:1+0 records out 2024-03-25T18:15:23.420 INFO:teuthology.orchestra.run.smithi097.stderr:512 bytes copied, 0.00023833 s, 2.1 MB/s 2024-03-25T18:15:23.421 DEBUG:teuthology.orchestra.run.smithi097:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-25T18:15:23.476 DEBUG:teuthology.orchestra.run.smithi097:> stat /dev/vg_nvme/lv_2 2024-03-25T18:15:23.531 INFO:teuthology.orchestra.run.smithi097.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-25T18:15:23.531 INFO:teuthology.orchestra.run.smithi097.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:15:23.531 INFO:teuthology.orchestra.run.smithi097.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-03-25T18:15:23.532 INFO:teuthology.orchestra.run.smithi097.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:15:23.532 INFO:teuthology.orchestra.run.smithi097.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:15:23.532 INFO:teuthology.orchestra.run.smithi097.stdout:Access: 2024-03-25 16:29:15.638559847 +0000 2024-03-25T18:15:23.532 INFO:teuthology.orchestra.run.smithi097.stdout:Modify: 2024-03-25 16:19:18.184081096 +0000 2024-03-25T18:15:23.532 INFO:teuthology.orchestra.run.smithi097.stdout:Change: 2024-03-25 16:19:18.184081096 +0000 2024-03-25T18:15:23.532 INFO:teuthology.orchestra.run.smithi097.stdout: Birth: 2024-03-25 16:19:18.184081096 +0000 2024-03-25T18:15:23.532 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-25T18:15:23.597 INFO:teuthology.orchestra.run.smithi097.stderr:1+0 records in 2024-03-25T18:15:23.597 INFO:teuthology.orchestra.run.smithi097.stderr:1+0 records out 2024-03-25T18:15:23.597 INFO:teuthology.orchestra.run.smithi097.stderr:512 bytes copied, 0.000212098 s, 2.4 MB/s 2024-03-25T18:15:23.599 DEBUG:teuthology.orchestra.run.smithi097:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-25T18:15:23.654 DEBUG:teuthology.orchestra.run.smithi097:> stat /dev/vg_nvme/lv_3 2024-03-25T18:15:23.709 INFO:teuthology.orchestra.run.smithi097.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-25T18:15:23.709 INFO:teuthology.orchestra.run.smithi097.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:15:23.710 INFO:teuthology.orchestra.run.smithi097.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-03-25T18:15:23.710 INFO:teuthology.orchestra.run.smithi097.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:15:23.710 INFO:teuthology.orchestra.run.smithi097.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:15:23.710 INFO:teuthology.orchestra.run.smithi097.stdout:Access: 2024-03-25 16:29:15.638559847 +0000 2024-03-25T18:15:23.710 INFO:teuthology.orchestra.run.smithi097.stdout:Modify: 2024-03-25 16:19:18.509075468 +0000 2024-03-25T18:15:23.710 INFO:teuthology.orchestra.run.smithi097.stdout:Change: 2024-03-25 16:19:18.509075468 +0000 2024-03-25T18:15:23.710 INFO:teuthology.orchestra.run.smithi097.stdout: Birth: 2024-03-25 16:19:18.509075468 +0000 2024-03-25T18:15:23.710 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-25T18:15:23.775 INFO:teuthology.orchestra.run.smithi097.stderr:1+0 records in 2024-03-25T18:15:23.775 INFO:teuthology.orchestra.run.smithi097.stderr:1+0 records out 2024-03-25T18:15:23.775 INFO:teuthology.orchestra.run.smithi097.stderr:512 bytes copied, 0.000199387 s, 2.6 MB/s 2024-03-25T18:15:23.776 DEBUG:teuthology.orchestra.run.smithi097:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-25T18:15:23.832 DEBUG:teuthology.orchestra.run.smithi097:> stat /dev/vg_nvme/lv_4 2024-03-25T18:15:23.887 INFO:teuthology.orchestra.run.smithi097.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-25T18:15:23.887 INFO:teuthology.orchestra.run.smithi097.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:15:23.887 INFO:teuthology.orchestra.run.smithi097.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-03-25T18:15:23.887 INFO:teuthology.orchestra.run.smithi097.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:15:23.887 INFO:teuthology.orchestra.run.smithi097.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:15:23.887 INFO:teuthology.orchestra.run.smithi097.stdout:Access: 2024-03-25 16:29:15.639559829 +0000 2024-03-25T18:15:23.887 INFO:teuthology.orchestra.run.smithi097.stdout:Modify: 2024-03-25 16:19:18.853069512 +0000 2024-03-25T18:15:23.887 INFO:teuthology.orchestra.run.smithi097.stdout:Change: 2024-03-25 16:19:18.853069512 +0000 2024-03-25T18:15:23.887 INFO:teuthology.orchestra.run.smithi097.stdout: Birth: 2024-03-25 16:19:18.853069512 +0000 2024-03-25T18:15:23.888 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-25T18:15:23.952 INFO:teuthology.orchestra.run.smithi097.stderr:1+0 records in 2024-03-25T18:15:23.952 INFO:teuthology.orchestra.run.smithi097.stderr:1+0 records out 2024-03-25T18:15:23.952 INFO:teuthology.orchestra.run.smithi097.stderr:512 bytes copied, 0.000202672 s, 2.5 MB/s 2024-03-25T18:15:23.953 DEBUG:teuthology.orchestra.run.smithi097:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-25T18:15:24.008 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T18:15:24.008 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-25T18:15:24.008 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/scratch_devs of=/dev/stdout 2024-03-25T18:15:24.025 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-25T18:15:24.026 DEBUG:teuthology.orchestra.run.smithi098:> stat /dev/vg_nvme/lv_1 2024-03-25T18:15:24.082 INFO:teuthology.orchestra.run.smithi098.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-25T18:15:24.082 INFO:teuthology.orchestra.run.smithi098.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:15:24.082 INFO:teuthology.orchestra.run.smithi098.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-03-25T18:15:24.082 INFO:teuthology.orchestra.run.smithi098.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:15:24.082 INFO:teuthology.orchestra.run.smithi098.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:15:24.082 INFO:teuthology.orchestra.run.smithi098.stdout:Access: 2024-03-25 16:23:45.241791899 +0000 2024-03-25T18:15:24.082 INFO:teuthology.orchestra.run.smithi098.stdout:Modify: 2024-03-25 16:19:17.849658685 +0000 2024-03-25T18:15:24.082 INFO:teuthology.orchestra.run.smithi098.stdout:Change: 2024-03-25 16:19:17.849658685 +0000 2024-03-25T18:15:24.083 INFO:teuthology.orchestra.run.smithi098.stdout: Birth: 2024-03-25 16:19:17.849658685 +0000 2024-03-25T18:15:24.083 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-25T18:15:24.151 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records in 2024-03-25T18:15:24.151 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records out 2024-03-25T18:15:24.151 INFO:teuthology.orchestra.run.smithi098.stderr:512 bytes copied, 0.000277638 s, 1.8 MB/s 2024-03-25T18:15:24.152 DEBUG:teuthology.orchestra.run.smithi098:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-25T18:15:24.210 DEBUG:teuthology.orchestra.run.smithi098:> stat /dev/vg_nvme/lv_2 2024-03-25T18:15:24.266 INFO:teuthology.orchestra.run.smithi098.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-25T18:15:24.266 INFO:teuthology.orchestra.run.smithi098.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:15:24.267 INFO:teuthology.orchestra.run.smithi098.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-03-25T18:15:24.267 INFO:teuthology.orchestra.run.smithi098.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:15:24.267 INFO:teuthology.orchestra.run.smithi098.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:15:24.267 INFO:teuthology.orchestra.run.smithi098.stdout:Access: 2024-03-25 16:23:45.241791899 +0000 2024-03-25T18:15:24.267 INFO:teuthology.orchestra.run.smithi098.stdout:Modify: 2024-03-25 16:19:18.179654055 +0000 2024-03-25T18:15:24.267 INFO:teuthology.orchestra.run.smithi098.stdout:Change: 2024-03-25 16:19:18.179654055 +0000 2024-03-25T18:15:24.267 INFO:teuthology.orchestra.run.smithi098.stdout: Birth: 2024-03-25 16:19:18.179654055 +0000 2024-03-25T18:15:24.267 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-25T18:15:24.333 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records in 2024-03-25T18:15:24.333 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records out 2024-03-25T18:15:24.333 INFO:teuthology.orchestra.run.smithi098.stderr:512 bytes copied, 0.000223816 s, 2.3 MB/s 2024-03-25T18:15:24.334 DEBUG:teuthology.orchestra.run.smithi098:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-25T18:15:24.391 DEBUG:teuthology.orchestra.run.smithi098:> stat /dev/vg_nvme/lv_3 2024-03-25T18:15:24.446 INFO:teuthology.orchestra.run.smithi098.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-25T18:15:24.446 INFO:teuthology.orchestra.run.smithi098.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:15:24.446 INFO:teuthology.orchestra.run.smithi098.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-03-25T18:15:24.446 INFO:teuthology.orchestra.run.smithi098.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:15:24.446 INFO:teuthology.orchestra.run.smithi098.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:15:24.446 INFO:teuthology.orchestra.run.smithi098.stdout:Access: 2024-03-25 16:23:45.241791899 +0000 2024-03-25T18:15:24.447 INFO:teuthology.orchestra.run.smithi098.stdout:Modify: 2024-03-25 16:19:18.510649410 +0000 2024-03-25T18:15:24.447 INFO:teuthology.orchestra.run.smithi098.stdout:Change: 2024-03-25 16:19:18.510649410 +0000 2024-03-25T18:15:24.447 INFO:teuthology.orchestra.run.smithi098.stdout: Birth: 2024-03-25 16:19:18.510649410 +0000 2024-03-25T18:15:24.447 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-25T18:15:24.512 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records in 2024-03-25T18:15:24.512 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records out 2024-03-25T18:15:24.512 INFO:teuthology.orchestra.run.smithi098.stderr:512 bytes copied, 0.000196006 s, 2.6 MB/s 2024-03-25T18:15:24.513 DEBUG:teuthology.orchestra.run.smithi098:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-25T18:15:24.569 DEBUG:teuthology.orchestra.run.smithi098:> stat /dev/vg_nvme/lv_4 2024-03-25T18:15:24.624 INFO:teuthology.orchestra.run.smithi098.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-25T18:15:24.624 INFO:teuthology.orchestra.run.smithi098.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:15:24.625 INFO:teuthology.orchestra.run.smithi098.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-03-25T18:15:24.625 INFO:teuthology.orchestra.run.smithi098.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:15:24.625 INFO:teuthology.orchestra.run.smithi098.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:15:24.625 INFO:teuthology.orchestra.run.smithi098.stdout:Access: 2024-03-25 16:23:45.242791885 +0000 2024-03-25T18:15:24.625 INFO:teuthology.orchestra.run.smithi098.stdout:Modify: 2024-03-25 16:19:18.862644469 +0000 2024-03-25T18:15:24.625 INFO:teuthology.orchestra.run.smithi098.stdout:Change: 2024-03-25 16:19:18.862644469 +0000 2024-03-25T18:15:24.625 INFO:teuthology.orchestra.run.smithi098.stdout: Birth: 2024-03-25 16:19:18.862644469 +0000 2024-03-25T18:15:24.625 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-25T18:15:24.690 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records in 2024-03-25T18:15:24.690 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records out 2024-03-25T18:15:24.690 INFO:teuthology.orchestra.run.smithi098.stderr:512 bytes copied, 0.00019584 s, 2.6 MB/s 2024-03-25T18:15:24.691 DEBUG:teuthology.orchestra.run.smithi098:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-25T18:15:24.747 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T18:15:24.747 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi097.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi098.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-25T18:15:24.747 INFO:tasks.ceph:Generating config... 2024-03-25T18:15:24.748 INFO:tasks.ceph:[client] debug monc = 20 2024-03-25T18:15:24.748 INFO:tasks.ceph:[client] debug ms = 1 2024-03-25T18:15:24.748 INFO:tasks.ceph:[client] debug objecter = 20 2024-03-25T18:15:24.748 INFO:tasks.ceph:[client] debug rados = 20 2024-03-25T18:15:24.748 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-03-25T18:15:24.748 INFO:tasks.ceph:[global] mon election default strategy = 3 2024-03-25T18:15:24.748 INFO:tasks.ceph:[global] ms bind msgr2 = False 2024-03-25T18:15:24.748 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-03-25T18:15:24.748 INFO:tasks.ceph:[global] ms type = async 2024-03-25T18:15:24.749 INFO:tasks.ceph:[global] osd heartbeat grace = 80 2024-03-25T18:15:24.749 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-25T18:15:24.749 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-25T18:15:24.749 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-25T18:15:24.749 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-25T18:15:24.749 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-25T18:15:24.749 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-03-25T18:15:24.749 INFO:tasks.ceph:[mon] mon osd crush smoke test = False 2024-03-25T18:15:24.749 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-03-25T18:15:24.749 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-03-25T18:15:24.749 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-03-25T18:15:24.749 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-03-25T18:15:24.749 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2024-03-25T18:15:24.749 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-03-25T18:15:24.749 INFO:tasks.ceph:[osd] bdev async discard = True 2024-03-25T18:15:24.749 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-03-25T18:15:24.750 INFO:tasks.ceph:[osd] bluestore allocator = stupid 2024-03-25T18:15:24.750 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-25T18:15:24.750 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-25T18:15:24.750 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-03-25T18:15:24.750 INFO:tasks.ceph:[osd] debug bluefs = 1 2024-03-25T18:15:24.750 INFO:tasks.ceph:[osd] debug bluestore = 1 2024-03-25T18:15:24.750 INFO:tasks.ceph:[osd] debug monc = 20 2024-03-25T18:15:24.750 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-25T18:15:24.750 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-25T18:15:24.750 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-03-25T18:15:24.750 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-25T18:15:24.750 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-25T18:15:24.750 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-25T18:15:24.750 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2024-03-25T18:15:24.751 INFO:tasks.ceph:[osd] osd class default list = * 2024-03-25T18:15:24.751 INFO:tasks.ceph:[osd] osd class load list = * 2024-03-25T18:15:24.751 INFO:tasks.ceph:[osd] osd client watch timeout = 120 2024-03-25T18:15:24.751 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-03-25T18:15:24.751 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-03-25T18:15:24.751 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-25T18:15:24.751 INFO:tasks.ceph:[osd] osd fast shutdown = False 2024-03-25T18:15:24.751 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2024-03-25T18:15:24.751 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-03-25T18:15:24.751 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-03-25T18:15:24.751 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-25T18:15:24.751 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-03-25T18:15:24.751 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-03-25T18:15:24.751 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-03-25T18:15:24.751 INFO:tasks.ceph:Setting up mon.a... 2024-03-25T18:15:24.752 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-25T18:15:24.811 INFO:teuthology.orchestra.run.smithi097.stdout:creating /etc/ceph/ceph.keyring 2024-03-25T18:15:24.814 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-25T18:15:24.859 DEBUG:teuthology.orchestra.run.smithi097:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-25T18:15:24.886 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '[v1:172.21.15.97:6789]'), ('mon.c', '[v1:172.21.15.97:6790]'), ('mon.b', '[v1:172.21.15.98:6789]')] 2024-03-25T18:15:24.887 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '[v1:172.21.15.97:6789],[v1:172.21.15.97:6790],[v1:172.21.15.98:6789]', 'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-03-25T18:15:24.887 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-25T18:15:24.887 DEBUG:teuthology.orchestra.run.smithi097:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-25T18:15:24.942 DEBUG:teuthology.orchestra.run.smithi097:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --addv a '[v1:172.21.15.97:6789]' --addv c '[v1:172.21.15.97:6790]' --addv b '[v1:172.21.15.98:6789]' --print /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:15:25.026 INFO:teuthology.orchestra.run.smithi097.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:15:25.026 INFO:teuthology.orchestra.run.smithi097.stdout:monmaptool: generated fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:15:25.026 INFO:teuthology.orchestra.run.smithi097.stdout:setting min_mon_release = quincy 2024-03-25T18:15:25.027 INFO:teuthology.orchestra.run.smithi097.stdout:epoch 0 2024-03-25T18:15:25.027 INFO:teuthology.orchestra.run.smithi097.stdout:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:15:25.027 INFO:teuthology.orchestra.run.smithi097.stdout:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:15:25.027 INFO:teuthology.orchestra.run.smithi097.stdout:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:15:25.027 INFO:teuthology.orchestra.run.smithi097.stdout:min_mon_release 17 (quincy) 2024-03-25T18:15:25.027 INFO:teuthology.orchestra.run.smithi097.stdout:election_strategy: 3 2024-03-25T18:15:25.027 INFO:teuthology.orchestra.run.smithi097.stdout:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:15:25.027 INFO:teuthology.orchestra.run.smithi097.stdout:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:15:25.027 INFO:teuthology.orchestra.run.smithi097.stdout:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:15:25.027 INFO:teuthology.orchestra.run.smithi097.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-03-25T18:15:25.028 DEBUG:teuthology.orchestra.run.smithi097:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-25T18:15:25.042 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 4c3d472c-79b8-486b-9e3d-2738d4306754... 2024-03-25T18:15:25.043 DEBUG:teuthology.orchestra.run.smithi097:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-25T18:15:25.085 DEBUG:teuthology.orchestra.run.smithi098:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-25T18:15:25.133 INFO:teuthology.orchestra.run.smithi097.stdout:[global] 2024-03-25T18:15:25.133 INFO:teuthology.orchestra.run.smithi097.stdout: chdir = "" 2024-03-25T18:15:25.133 INFO:teuthology.orchestra.run.smithi097.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-25T18:15:25.133 INFO:teuthology.orchestra.run.smithi097.stdout: auth supported = cephx 2024-03-25T18:15:25.133 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T18:15:25.133 INFO:teuthology.orchestra.run.smithi097.stdout: filestore xattr use omap = true 2024-03-25T18:15:25.133 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T18:15:25.133 INFO:teuthology.orchestra.run.smithi097.stdout: mon clock drift allowed = 1.000 2024-03-25T18:15:25.134 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T18:15:25.134 INFO:teuthology.orchestra.run.smithi097.stdout: osd crush chooseleaf type = 0 2024-03-25T18:15:25.134 INFO:teuthology.orchestra.run.smithi097.stdout: auth debug = true 2024-03-25T18:15:25.134 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T18:15:25.134 INFO:teuthology.orchestra.run.smithi097.stdout: ms die on old message = true 2024-03-25T18:15:25.134 INFO:teuthology.orchestra.run.smithi097.stdout: ms die on bug = true 2024-03-25T18:15:25.134 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T18:15:25.134 INFO:teuthology.orchestra.run.smithi097.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-25T18:15:25.134 INFO:teuthology.orchestra.run.smithi097.stdout: mon pg warn max object skew = 0 2024-03-25T18:15:25.134 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T18:15:25.134 INFO:teuthology.orchestra.run.smithi097.stdout: # disable pg_autoscaler by default for new pools 2024-03-25T18:15:25.134 INFO:teuthology.orchestra.run.smithi097.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-25T18:15:25.135 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T18:15:25.135 INFO:teuthology.orchestra.run.smithi097.stdout: osd pool default size = 2 2024-03-25T18:15:25.135 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T18:15:25.135 INFO:teuthology.orchestra.run.smithi097.stdout: mon osd allow primary affinity = true 2024-03-25T18:15:25.135 INFO:teuthology.orchestra.run.smithi097.stdout: mon osd allow pg remap = true 2024-03-25T18:15:25.135 INFO:teuthology.orchestra.run.smithi097.stdout: mon warn on legacy crush tunables = false 2024-03-25T18:15:25.135 INFO:teuthology.orchestra.run.smithi097.stdout: mon warn on crush straw calc version zero = false 2024-03-25T18:15:25.135 INFO:teuthology.orchestra.run.smithi097.stdout: mon warn on no sortbitwise = false 2024-03-25T18:15:25.135 INFO:teuthology.orchestra.run.smithi097.stdout: mon warn on osd down out interval zero = false 2024-03-25T18:15:25.135 INFO:teuthology.orchestra.run.smithi097.stdout: mon warn on too few osds = false 2024-03-25T18:15:25.135 INFO:teuthology.orchestra.run.smithi097.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-25T18:15:25.135 INFO:teuthology.orchestra.run.smithi097.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-25T18:15:25.135 INFO:teuthology.orchestra.run.smithi097.stdout: mon_allow_pool_size_one = true 2024-03-25T18:15:25.135 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T18:15:25.136 INFO:teuthology.orchestra.run.smithi097.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-25T18:15:25.136 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T18:15:25.136 INFO:teuthology.orchestra.run.smithi097.stdout: osd default data pool replay window = 5 2024-03-25T18:15:25.136 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T18:15:25.136 INFO:teuthology.orchestra.run.smithi097.stdout: mon allow pool delete = true 2024-03-25T18:15:25.136 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T18:15:25.136 INFO:teuthology.orchestra.run.smithi097.stdout: mon cluster log file level = debug 2024-03-25T18:15:25.136 INFO:teuthology.orchestra.run.smithi097.stdout: debug asserts on shutdown = true 2024-03-25T18:15:25.136 INFO:teuthology.orchestra.run.smithi097.stdout: mon health detail to clog = false 2024-03-25T18:15:25.136 INFO:teuthology.orchestra.run.smithi097.stdout: mon host = "[v1:172.21.15.97:6789],[v1:172.21.15.97:6790],[v1:172.21.15.98:6789]" 2024-03-25T18:15:25.136 INFO:teuthology.orchestra.run.smithi097.stdout: mon client directed command retry = 5 2024-03-25T18:15:25.136 INFO:teuthology.orchestra.run.smithi097.stdout: mon election default strategy = 3 2024-03-25T18:15:25.136 INFO:teuthology.orchestra.run.smithi097.stdout: ms bind msgr2 = False 2024-03-25T18:15:25.137 INFO:teuthology.orchestra.run.smithi097.stdout: ms inject socket failures = 5000 2024-03-25T18:15:25.137 INFO:teuthology.orchestra.run.smithi097.stdout: ms type = async 2024-03-25T18:15:25.137 INFO:teuthology.orchestra.run.smithi097.stdout: osd heartbeat grace = 80 2024-03-25T18:15:25.137 INFO:teuthology.orchestra.run.smithi097.stdout: fsid = 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:15:25.137 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T18:15:25.137 INFO:teuthology.orchestra.run.smithi097.stdout:[osd] 2024-03-25T18:15:25.137 INFO:teuthology.orchestra.run.smithi097.stdout: osd journal size = 100 2024-03-25T18:15:25.137 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T18:15:25.137 INFO:teuthology.orchestra.run.smithi097.stdout: osd scrub load threshold = 5.0 2024-03-25T18:15:25.137 INFO:teuthology.orchestra.run.smithi097.stdout: osd scrub max interval = 600 2024-03-25T18:15:25.137 INFO:teuthology.orchestra.run.smithi097.stdout: osd mclock profile = high_recovery_ops 2024-03-25T18:15:25.137 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T18:15:25.137 INFO:teuthology.orchestra.run.smithi097.stdout: osd recover clone overlap = true 2024-03-25T18:15:25.137 INFO:teuthology.orchestra.run.smithi097.stdout: osd recovery max chunk = 1048576 2024-03-25T18:15:25.137 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T18:15:25.138 INFO:teuthology.orchestra.run.smithi097.stdout: osd debug shutdown = true 2024-03-25T18:15:25.138 INFO:teuthology.orchestra.run.smithi097.stdout: osd debug op order = true 2024-03-25T18:15:25.138 INFO:teuthology.orchestra.run.smithi097.stdout: osd debug verify stray on activate = true 2024-03-25T18:15:25.138 INFO:teuthology.orchestra.run.smithi097.stdout: osd debug trim objects = true 2024-03-25T18:15:25.138 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T18:15:25.138 INFO:teuthology.orchestra.run.smithi097.stdout: osd open classes on start = true 2024-03-25T18:15:25.138 INFO:teuthology.orchestra.run.smithi097.stdout: osd debug pg log writeout = true 2024-03-25T18:15:25.138 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T18:15:25.138 INFO:teuthology.orchestra.run.smithi097.stdout: osd deep scrub update digest min age = 30 2024-03-25T18:15:25.138 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T18:15:25.138 INFO:teuthology.orchestra.run.smithi097.stdout: osd map max advance = 10 2024-03-25T18:15:25.138 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T18:15:25.138 INFO:teuthology.orchestra.run.smithi097.stdout: journal zero on create = true 2024-03-25T18:15:25.138 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T18:15:25.139 INFO:teuthology.orchestra.run.smithi097.stdout: filestore ondisk finisher threads = 3 2024-03-25T18:15:25.139 INFO:teuthology.orchestra.run.smithi097.stdout: filestore apply finisher threads = 3 2024-03-25T18:15:25.139 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T18:15:25.139 INFO:teuthology.orchestra.run.smithi097.stdout: bdev debug aio = true 2024-03-25T18:15:25.139 INFO:teuthology.orchestra.run.smithi097.stdout: osd debug misdirected ops = true 2024-03-25T18:15:25.139 INFO:teuthology.orchestra.run.smithi097.stdout: bdev async discard = True 2024-03-25T18:15:25.139 INFO:teuthology.orchestra.run.smithi097.stdout: bdev enable discard = True 2024-03-25T18:15:25.139 INFO:teuthology.orchestra.run.smithi097.stdout: bluestore allocator = stupid 2024-03-25T18:15:25.139 INFO:teuthology.orchestra.run.smithi097.stdout: bluestore block size = 96636764160 2024-03-25T18:15:25.139 INFO:teuthology.orchestra.run.smithi097.stdout: bluestore fsck on mount = True 2024-03-25T18:15:25.139 INFO:teuthology.orchestra.run.smithi097.stdout: bluestore zero block detection = True 2024-03-25T18:15:25.139 INFO:teuthology.orchestra.run.smithi097.stdout: debug bluefs = 1 2024-03-25T18:15:25.139 INFO:teuthology.orchestra.run.smithi097.stdout: debug bluestore = 1 2024-03-25T18:15:25.139 INFO:teuthology.orchestra.run.smithi097.stdout: debug monc = 20 2024-03-25T18:15:25.140 INFO:teuthology.orchestra.run.smithi097.stdout: debug ms = 1 2024-03-25T18:15:25.140 INFO:teuthology.orchestra.run.smithi097.stdout: debug osd = 20 2024-03-25T18:15:25.140 INFO:teuthology.orchestra.run.smithi097.stdout: debug rocksdb = 10 2024-03-25T18:15:25.140 INFO:teuthology.orchestra.run.smithi097.stdout: mon osd backfillfull_ratio = 0.85 2024-03-25T18:15:25.140 INFO:teuthology.orchestra.run.smithi097.stdout: mon osd full ratio = 0.9 2024-03-25T18:15:25.140 INFO:teuthology.orchestra.run.smithi097.stdout: mon osd nearfull ratio = 0.8 2024-03-25T18:15:25.140 INFO:teuthology.orchestra.run.smithi097.stdout: osd blocked scrub grace period = 3600 2024-03-25T18:15:25.140 INFO:teuthology.orchestra.run.smithi097.stdout: osd class default list = * 2024-03-25T18:15:25.140 INFO:teuthology.orchestra.run.smithi097.stdout: osd class load list = * 2024-03-25T18:15:25.140 INFO:teuthology.orchestra.run.smithi097.stdout: osd client watch timeout = 120 2024-03-25T18:15:25.140 INFO:teuthology.orchestra.run.smithi097.stdout: osd debug verify cached snaps = True 2024-03-25T18:15:25.140 INFO:teuthology.orchestra.run.smithi097.stdout: osd debug verify missing on start = True 2024-03-25T18:15:25.140 INFO:teuthology.orchestra.run.smithi097.stdout: osd failsafe full ratio = 0.95 2024-03-25T18:15:25.140 INFO:teuthology.orchestra.run.smithi097.stdout: osd fast shutdown = False 2024-03-25T18:15:25.141 INFO:teuthology.orchestra.run.smithi097.stdout: osd max markdown count = 1000 2024-03-25T18:15:25.141 INFO:teuthology.orchestra.run.smithi097.stdout: osd mclock override recovery settings = True 2024-03-25T18:15:25.141 INFO:teuthology.orchestra.run.smithi097.stdout: osd objectstore = bluestore 2024-03-25T18:15:25.141 INFO:teuthology.orchestra.run.smithi097.stdout: osd op queue = debug_random 2024-03-25T18:15:25.141 INFO:teuthology.orchestra.run.smithi097.stdout: osd op queue cut off = debug_random 2024-03-25T18:15:25.141 INFO:teuthology.orchestra.run.smithi097.stdout: osd shutdown pgref assert = True 2024-03-25T18:15:25.141 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T18:15:25.141 INFO:teuthology.orchestra.run.smithi097.stdout:[mgr] 2024-03-25T18:15:25.141 INFO:teuthology.orchestra.run.smithi097.stdout: debug ms = 1 2024-03-25T18:15:25.141 INFO:teuthology.orchestra.run.smithi097.stdout: debug mgr = 20 2024-03-25T18:15:25.141 INFO:teuthology.orchestra.run.smithi097.stdout: debug mon = 20 2024-03-25T18:15:25.141 INFO:teuthology.orchestra.run.smithi097.stdout: debug auth = 20 2024-03-25T18:15:25.141 INFO:teuthology.orchestra.run.smithi097.stdout: mon reweight min pgs per osd = 4 2024-03-25T18:15:25.141 INFO:teuthology.orchestra.run.smithi097.stdout: mon reweight min bytes per osd = 10 2024-03-25T18:15:25.141 INFO:teuthology.orchestra.run.smithi097.stdout: mgr/telemetry/nag = false 2024-03-25T18:15:25.142 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T18:15:25.142 INFO:teuthology.orchestra.run.smithi097.stdout:[mon] 2024-03-25T18:15:25.142 INFO:teuthology.orchestra.run.smithi097.stdout: debug ms = 1 2024-03-25T18:15:25.142 INFO:teuthology.orchestra.run.smithi097.stdout: debug mon = 20 2024-03-25T18:15:25.142 INFO:teuthology.orchestra.run.smithi097.stdout: debug paxos = 20 2024-03-25T18:15:25.142 INFO:teuthology.orchestra.run.smithi097.stdout: debug auth = 20 2024-03-25T18:15:25.142 INFO:teuthology.orchestra.run.smithi097.stdout: mon data avail warn = 5 2024-03-25T18:15:25.142 INFO:teuthology.orchestra.run.smithi097.stdout: mon mgr mkfs grace = 240 2024-03-25T18:15:25.142 INFO:teuthology.orchestra.run.smithi097.stdout: mon reweight min pgs per osd = 4 2024-03-25T18:15:25.142 INFO:teuthology.orchestra.run.smithi097.stdout: mon osd reporter subtree level = osd 2024-03-25T18:15:25.142 INFO:teuthology.orchestra.run.smithi097.stdout: mon osd prime pg temp = true 2024-03-25T18:15:25.142 INFO:teuthology.orchestra.run.smithi097.stdout: mon reweight min bytes per osd = 10 2024-03-25T18:15:25.142 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T18:15:25.143 INFO:teuthology.orchestra.run.smithi097.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-25T18:15:25.143 INFO:teuthology.orchestra.run.smithi097.stdout: auth mon ticket ttl = 660 # 11m 2024-03-25T18:15:25.143 INFO:teuthology.orchestra.run.smithi097.stdout: auth service ticket ttl = 240 # 4m 2024-03-25T18:15:25.143 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T18:15:25.143 INFO:teuthology.orchestra.run.smithi097.stdout: # don't complain about insecure global_id in the test suite 2024-03-25T18:15:25.143 INFO:teuthology.orchestra.run.smithi097.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-25T18:15:25.143 INFO:teuthology.orchestra.run.smithi097.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-25T18:15:25.143 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T18:15:25.143 INFO:teuthology.orchestra.run.smithi097.stdout: # 1m isn't quite enough 2024-03-25T18:15:25.143 INFO:teuthology.orchestra.run.smithi097.stdout: mon_down_mkfs_grace = 2m 2024-03-25T18:15:25.143 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T18:15:25.143 INFO:teuthology.orchestra.run.smithi097.stdout: mon_warn_on_filestore_osds = false 2024-03-25T18:15:25.143 INFO:teuthology.orchestra.run.smithi097.stdout: mon min osdmap epochs = 50 2024-03-25T18:15:25.143 INFO:teuthology.orchestra.run.smithi097.stdout: mon osd crush smoke test = False 2024-03-25T18:15:25.143 INFO:teuthology.orchestra.run.smithi097.stdout: mon osdmap full prune interval = 2 2024-03-25T18:15:25.144 INFO:teuthology.orchestra.run.smithi097.stdout: mon osdmap full prune min = 15 2024-03-25T18:15:25.144 INFO:teuthology.orchestra.run.smithi097.stdout: mon osdmap full prune txsize = 2 2024-03-25T18:15:25.144 INFO:teuthology.orchestra.run.smithi097.stdout: mon scrub interval = 300 2024-03-25T18:15:25.144 INFO:teuthology.orchestra.run.smithi097.stdout: mon warn on pool no app = False 2024-03-25T18:15:25.144 INFO:teuthology.orchestra.run.smithi097.stdout: paxos service trim min = 10 2024-03-25T18:15:25.144 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T18:15:25.144 INFO:teuthology.orchestra.run.smithi097.stdout:[client] 2024-03-25T18:15:25.144 INFO:teuthology.orchestra.run.smithi097.stdout: rgw cache enabled = true 2024-03-25T18:15:25.144 INFO:teuthology.orchestra.run.smithi097.stdout: rgw enable ops log = true 2024-03-25T18:15:25.144 INFO:teuthology.orchestra.run.smithi097.stdout: rgw enable usage log = true 2024-03-25T18:15:25.144 INFO:teuthology.orchestra.run.smithi097.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-25T18:15:25.144 INFO:teuthology.orchestra.run.smithi097.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-25T18:15:25.144 INFO:teuthology.orchestra.run.smithi097.stdout: debug monc = 20 2024-03-25T18:15:25.144 INFO:teuthology.orchestra.run.smithi097.stdout: debug ms = 1 2024-03-25T18:15:25.144 INFO:teuthology.orchestra.run.smithi097.stdout: debug objecter = 20 2024-03-25T18:15:25.145 INFO:teuthology.orchestra.run.smithi097.stdout: debug rados = 20 2024-03-25T18:15:25.145 INFO:teuthology.orchestra.run.smithi097.stdout:[mon.a] 2024-03-25T18:15:25.145 INFO:teuthology.orchestra.run.smithi097.stdout:[mon.c] 2024-03-25T18:15:25.145 INFO:teuthology.orchestra.run.smithi097.stdout:[mon.b] 2024-03-25T18:15:25.145 INFO:teuthology.orchestra.run.smithi098.stdout:[global] 2024-03-25T18:15:25.145 INFO:teuthology.orchestra.run.smithi098.stdout: chdir = "" 2024-03-25T18:15:25.145 INFO:teuthology.orchestra.run.smithi098.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-25T18:15:25.145 INFO:teuthology.orchestra.run.smithi098.stdout: auth supported = cephx 2024-03-25T18:15:25.145 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T18:15:25.145 INFO:teuthology.orchestra.run.smithi098.stdout: filestore xattr use omap = true 2024-03-25T18:15:25.146 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T18:15:25.146 INFO:teuthology.orchestra.run.smithi098.stdout: mon clock drift allowed = 1.000 2024-03-25T18:15:25.146 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T18:15:25.146 INFO:teuthology.orchestra.run.smithi098.stdout: osd crush chooseleaf type = 0 2024-03-25T18:15:25.146 INFO:teuthology.orchestra.run.smithi098.stdout: auth debug = true 2024-03-25T18:15:25.146 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T18:15:25.146 INFO:teuthology.orchestra.run.smithi098.stdout: ms die on old message = true 2024-03-25T18:15:25.146 INFO:teuthology.orchestra.run.smithi098.stdout: ms die on bug = true 2024-03-25T18:15:25.146 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T18:15:25.146 INFO:teuthology.orchestra.run.smithi098.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-25T18:15:25.146 INFO:teuthology.orchestra.run.smithi098.stdout: mon pg warn max object skew = 0 2024-03-25T18:15:25.146 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T18:15:25.146 INFO:teuthology.orchestra.run.smithi098.stdout: # disable pg_autoscaler by default for new pools 2024-03-25T18:15:25.147 INFO:teuthology.orchestra.run.smithi098.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-25T18:15:25.147 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T18:15:25.147 INFO:teuthology.orchestra.run.smithi098.stdout: osd pool default size = 2 2024-03-25T18:15:25.147 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T18:15:25.147 INFO:teuthology.orchestra.run.smithi098.stdout: mon osd allow primary affinity = true 2024-03-25T18:15:25.147 INFO:teuthology.orchestra.run.smithi098.stdout: mon osd allow pg remap = true 2024-03-25T18:15:25.147 INFO:teuthology.orchestra.run.smithi098.stdout: mon warn on legacy crush tunables = false 2024-03-25T18:15:25.147 INFO:teuthology.orchestra.run.smithi098.stdout: mon warn on crush straw calc version zero = false 2024-03-25T18:15:25.147 INFO:teuthology.orchestra.run.smithi098.stdout: mon warn on no sortbitwise = false 2024-03-25T18:15:25.147 INFO:teuthology.orchestra.run.smithi098.stdout: mon warn on osd down out interval zero = false 2024-03-25T18:15:25.147 INFO:teuthology.orchestra.run.smithi098.stdout: mon warn on too few osds = false 2024-03-25T18:15:25.147 INFO:teuthology.orchestra.run.smithi098.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-25T18:15:25.147 INFO:teuthology.orchestra.run.smithi098.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-25T18:15:25.147 INFO:teuthology.orchestra.run.smithi098.stdout: mon_allow_pool_size_one = true 2024-03-25T18:15:25.147 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T18:15:25.148 INFO:teuthology.orchestra.run.smithi098.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-25T18:15:25.148 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T18:15:25.148 INFO:teuthology.orchestra.run.smithi098.stdout: osd default data pool replay window = 5 2024-03-25T18:15:25.148 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T18:15:25.148 INFO:teuthology.orchestra.run.smithi098.stdout: mon allow pool delete = true 2024-03-25T18:15:25.148 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T18:15:25.148 INFO:teuthology.orchestra.run.smithi098.stdout: mon cluster log file level = debug 2024-03-25T18:15:25.148 INFO:teuthology.orchestra.run.smithi098.stdout: debug asserts on shutdown = true 2024-03-25T18:15:25.148 INFO:teuthology.orchestra.run.smithi098.stdout: mon health detail to clog = false 2024-03-25T18:15:25.148 INFO:teuthology.orchestra.run.smithi098.stdout: mon host = "[v1:172.21.15.97:6789],[v1:172.21.15.97:6790],[v1:172.21.15.98:6789]" 2024-03-25T18:15:25.148 INFO:teuthology.orchestra.run.smithi098.stdout: mon client directed command retry = 5 2024-03-25T18:15:25.148 INFO:teuthology.orchestra.run.smithi098.stdout: mon election default strategy = 3 2024-03-25T18:15:25.148 INFO:teuthology.orchestra.run.smithi098.stdout: ms bind msgr2 = False 2024-03-25T18:15:25.148 INFO:teuthology.orchestra.run.smithi098.stdout: ms inject socket failures = 5000 2024-03-25T18:15:25.149 INFO:teuthology.orchestra.run.smithi098.stdout: ms type = async 2024-03-25T18:15:25.149 INFO:teuthology.orchestra.run.smithi098.stdout: osd heartbeat grace = 80 2024-03-25T18:15:25.149 INFO:teuthology.orchestra.run.smithi098.stdout: fsid = 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:15:25.149 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T18:15:25.149 INFO:teuthology.orchestra.run.smithi098.stdout:[osd] 2024-03-25T18:15:25.149 INFO:teuthology.orchestra.run.smithi098.stdout: osd journal size = 100 2024-03-25T18:15:25.149 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T18:15:25.149 INFO:teuthology.orchestra.run.smithi098.stdout: osd scrub load threshold = 5.0 2024-03-25T18:15:25.149 INFO:teuthology.orchestra.run.smithi098.stdout: osd scrub max interval = 600 2024-03-25T18:15:25.149 INFO:teuthology.orchestra.run.smithi098.stdout: osd mclock profile = high_recovery_ops 2024-03-25T18:15:25.149 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T18:15:25.149 INFO:teuthology.orchestra.run.smithi098.stdout: osd recover clone overlap = true 2024-03-25T18:15:25.149 INFO:teuthology.orchestra.run.smithi098.stdout: osd recovery max chunk = 1048576 2024-03-25T18:15:25.149 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T18:15:25.149 INFO:teuthology.orchestra.run.smithi098.stdout: osd debug shutdown = true 2024-03-25T18:15:25.150 INFO:teuthology.orchestra.run.smithi098.stdout: osd debug op order = true 2024-03-25T18:15:25.150 INFO:teuthology.orchestra.run.smithi098.stdout: osd debug verify stray on activate = true 2024-03-25T18:15:25.150 INFO:teuthology.orchestra.run.smithi098.stdout: osd debug trim objects = true 2024-03-25T18:15:25.150 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T18:15:25.150 INFO:teuthology.orchestra.run.smithi098.stdout: osd open classes on start = true 2024-03-25T18:15:25.150 INFO:teuthology.orchestra.run.smithi098.stdout: osd debug pg log writeout = true 2024-03-25T18:15:25.150 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T18:15:25.150 INFO:teuthology.orchestra.run.smithi098.stdout: osd deep scrub update digest min age = 30 2024-03-25T18:15:25.150 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T18:15:25.150 INFO:teuthology.orchestra.run.smithi098.stdout: osd map max advance = 10 2024-03-25T18:15:25.150 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T18:15:25.150 INFO:teuthology.orchestra.run.smithi098.stdout: journal zero on create = true 2024-03-25T18:15:25.150 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T18:15:25.150 INFO:teuthology.orchestra.run.smithi098.stdout: filestore ondisk finisher threads = 3 2024-03-25T18:15:25.150 INFO:teuthology.orchestra.run.smithi098.stdout: filestore apply finisher threads = 3 2024-03-25T18:15:25.151 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T18:15:25.151 INFO:teuthology.orchestra.run.smithi098.stdout: bdev debug aio = true 2024-03-25T18:15:25.151 INFO:teuthology.orchestra.run.smithi098.stdout: osd debug misdirected ops = true 2024-03-25T18:15:25.151 INFO:teuthology.orchestra.run.smithi098.stdout: bdev async discard = True 2024-03-25T18:15:25.151 INFO:teuthology.orchestra.run.smithi098.stdout: bdev enable discard = True 2024-03-25T18:15:25.151 INFO:teuthology.orchestra.run.smithi098.stdout: bluestore allocator = stupid 2024-03-25T18:15:25.151 INFO:teuthology.orchestra.run.smithi098.stdout: bluestore block size = 96636764160 2024-03-25T18:15:25.151 INFO:teuthology.orchestra.run.smithi098.stdout: bluestore fsck on mount = True 2024-03-25T18:15:25.151 INFO:teuthology.orchestra.run.smithi098.stdout: bluestore zero block detection = True 2024-03-25T18:15:25.151 INFO:teuthology.orchestra.run.smithi098.stdout: debug bluefs = 1 2024-03-25T18:15:25.151 INFO:teuthology.orchestra.run.smithi098.stdout: debug bluestore = 1 2024-03-25T18:15:25.151 INFO:teuthology.orchestra.run.smithi098.stdout: debug monc = 20 2024-03-25T18:15:25.151 INFO:teuthology.orchestra.run.smithi098.stdout: debug ms = 1 2024-03-25T18:15:25.151 INFO:teuthology.orchestra.run.smithi098.stdout: debug osd = 20 2024-03-25T18:15:25.151 INFO:teuthology.orchestra.run.smithi098.stdout: debug rocksdb = 10 2024-03-25T18:15:25.152 INFO:teuthology.orchestra.run.smithi098.stdout: mon osd backfillfull_ratio = 0.85 2024-03-25T18:15:25.152 INFO:teuthology.orchestra.run.smithi098.stdout: mon osd full ratio = 0.9 2024-03-25T18:15:25.152 INFO:teuthology.orchestra.run.smithi098.stdout: mon osd nearfull ratio = 0.8 2024-03-25T18:15:25.152 INFO:teuthology.orchestra.run.smithi098.stdout: osd blocked scrub grace period = 3600 2024-03-25T18:15:25.152 INFO:teuthology.orchestra.run.smithi098.stdout: osd class default list = * 2024-03-25T18:15:25.152 INFO:teuthology.orchestra.run.smithi098.stdout: osd class load list = * 2024-03-25T18:15:25.152 INFO:teuthology.orchestra.run.smithi098.stdout: osd client watch timeout = 120 2024-03-25T18:15:25.152 INFO:teuthology.orchestra.run.smithi098.stdout: osd debug verify cached snaps = True 2024-03-25T18:15:25.152 INFO:teuthology.orchestra.run.smithi098.stdout: osd debug verify missing on start = True 2024-03-25T18:15:25.152 INFO:teuthology.orchestra.run.smithi098.stdout: osd failsafe full ratio = 0.95 2024-03-25T18:15:25.152 INFO:teuthology.orchestra.run.smithi098.stdout: osd fast shutdown = False 2024-03-25T18:15:25.152 INFO:teuthology.orchestra.run.smithi098.stdout: osd max markdown count = 1000 2024-03-25T18:15:25.152 INFO:teuthology.orchestra.run.smithi098.stdout: osd mclock override recovery settings = True 2024-03-25T18:15:25.152 INFO:teuthology.orchestra.run.smithi098.stdout: osd objectstore = bluestore 2024-03-25T18:15:25.153 INFO:teuthology.orchestra.run.smithi098.stdout: osd op queue = debug_random 2024-03-25T18:15:25.153 INFO:teuthology.orchestra.run.smithi098.stdout: osd op queue cut off = debug_random 2024-03-25T18:15:25.153 INFO:teuthology.orchestra.run.smithi098.stdout: osd shutdown pgref assert = True 2024-03-25T18:15:25.153 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T18:15:25.153 INFO:teuthology.orchestra.run.smithi098.stdout:[mgr] 2024-03-25T18:15:25.153 INFO:teuthology.orchestra.run.smithi098.stdout: debug ms = 1 2024-03-25T18:15:25.153 INFO:teuthology.orchestra.run.smithi098.stdout: debug mgr = 20 2024-03-25T18:15:25.153 INFO:teuthology.orchestra.run.smithi098.stdout: debug mon = 20 2024-03-25T18:15:25.153 INFO:teuthology.orchestra.run.smithi098.stdout: debug auth = 20 2024-03-25T18:15:25.153 INFO:teuthology.orchestra.run.smithi098.stdout: mon reweight min pgs per osd = 4 2024-03-25T18:15:25.153 INFO:teuthology.orchestra.run.smithi098.stdout: mon reweight min bytes per osd = 10 2024-03-25T18:15:25.153 INFO:teuthology.orchestra.run.smithi098.stdout: mgr/telemetry/nag = false 2024-03-25T18:15:25.153 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T18:15:25.154 INFO:teuthology.orchestra.run.smithi098.stdout:[mon] 2024-03-25T18:15:25.154 INFO:teuthology.orchestra.run.smithi098.stdout: debug ms = 1 2024-03-25T18:15:25.154 INFO:teuthology.orchestra.run.smithi098.stdout: debug mon = 20 2024-03-25T18:15:25.154 INFO:teuthology.orchestra.run.smithi098.stdout: debug paxos = 20 2024-03-25T18:15:25.154 INFO:teuthology.orchestra.run.smithi098.stdout: debug auth = 20 2024-03-25T18:15:25.154 INFO:teuthology.orchestra.run.smithi098.stdout: mon data avail warn = 5 2024-03-25T18:15:25.154 INFO:teuthology.orchestra.run.smithi098.stdout: mon mgr mkfs grace = 240 2024-03-25T18:15:25.154 INFO:teuthology.orchestra.run.smithi098.stdout: mon reweight min pgs per osd = 4 2024-03-25T18:15:25.154 INFO:teuthology.orchestra.run.smithi098.stdout: mon osd reporter subtree level = osd 2024-03-25T18:15:25.154 INFO:teuthology.orchestra.run.smithi098.stdout: mon osd prime pg temp = true 2024-03-25T18:15:25.154 INFO:teuthology.orchestra.run.smithi098.stdout: mon reweight min bytes per osd = 10 2024-03-25T18:15:25.154 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T18:15:25.154 INFO:teuthology.orchestra.run.smithi098.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-25T18:15:25.155 INFO:teuthology.orchestra.run.smithi098.stdout: auth mon ticket ttl = 660 # 11m 2024-03-25T18:15:25.155 INFO:teuthology.orchestra.run.smithi098.stdout: auth service ticket ttl = 240 # 4m 2024-03-25T18:15:25.155 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T18:15:25.155 INFO:teuthology.orchestra.run.smithi098.stdout: # don't complain about insecure global_id in the test suite 2024-03-25T18:15:25.155 INFO:teuthology.orchestra.run.smithi098.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-25T18:15:25.155 INFO:teuthology.orchestra.run.smithi098.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-25T18:15:25.155 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T18:15:25.155 INFO:teuthology.orchestra.run.smithi098.stdout: # 1m isn't quite enough 2024-03-25T18:15:25.155 INFO:teuthology.orchestra.run.smithi098.stdout: mon_down_mkfs_grace = 2m 2024-03-25T18:15:25.155 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T18:15:25.155 INFO:teuthology.orchestra.run.smithi098.stdout: mon_warn_on_filestore_osds = false 2024-03-25T18:15:25.155 INFO:teuthology.orchestra.run.smithi098.stdout: mon min osdmap epochs = 50 2024-03-25T18:15:25.155 INFO:teuthology.orchestra.run.smithi098.stdout: mon osd crush smoke test = False 2024-03-25T18:15:25.156 INFO:teuthology.orchestra.run.smithi098.stdout: mon osdmap full prune interval = 2 2024-03-25T18:15:25.156 INFO:teuthology.orchestra.run.smithi098.stdout: mon osdmap full prune min = 15 2024-03-25T18:15:25.156 INFO:teuthology.orchestra.run.smithi098.stdout: mon osdmap full prune txsize = 2 2024-03-25T18:15:25.156 INFO:teuthology.orchestra.run.smithi098.stdout: mon scrub interval = 300 2024-03-25T18:15:25.156 INFO:teuthology.orchestra.run.smithi098.stdout: mon warn on pool no app = False 2024-03-25T18:15:25.156 INFO:teuthology.orchestra.run.smithi098.stdout: paxos service trim min = 10 2024-03-25T18:15:25.156 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T18:15:25.156 INFO:teuthology.orchestra.run.smithi098.stdout:[client] 2024-03-25T18:15:25.156 INFO:teuthology.orchestra.run.smithi098.stdout: rgw cache enabled = true 2024-03-25T18:15:25.156 INFO:teuthology.orchestra.run.smithi098.stdout: rgw enable ops log = true 2024-03-25T18:15:25.156 INFO:teuthology.orchestra.run.smithi098.stdout: rgw enable usage log = true 2024-03-25T18:15:25.156 INFO:teuthology.orchestra.run.smithi098.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-25T18:15:25.156 INFO:teuthology.orchestra.run.smithi098.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-25T18:15:25.156 INFO:teuthology.orchestra.run.smithi098.stdout: debug monc = 20 2024-03-25T18:15:25.156 INFO:teuthology.orchestra.run.smithi098.stdout: debug ms = 1 2024-03-25T18:15:25.157 INFO:teuthology.orchestra.run.smithi098.stdout: debug objecter = 20 2024-03-25T18:15:25.157 INFO:teuthology.orchestra.run.smithi098.stdout: debug rados = 20 2024-03-25T18:15:25.157 INFO:teuthology.orchestra.run.smithi098.stdout:[mon.a] 2024-03-25T18:15:25.157 INFO:teuthology.orchestra.run.smithi098.stdout:[mon.c] 2024-03-25T18:15:25.157 INFO:teuthology.orchestra.run.smithi098.stdout:[mon.b] 2024-03-25T18:15:25.158 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-25T18:15:25.158 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-25T18:15:25.213 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-25T18:15:25.213 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-25T18:15:25.213 DEBUG:teuthology.orchestra.run.smithi097:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-25T18:15:25.227 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-25T18:15:25.227 DEBUG:teuthology.orchestra.run.smithi097:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-25T18:15:25.282 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi097.front.sepia.ceph.com 2024-03-25T18:15:25.282 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-25T18:15:25.282 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-25T18:15:25.282 DEBUG:teuthology.orchestra.run.smithi097:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-25T18:15:25.361 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-25T18:15:25.361 DEBUG:teuthology.orchestra.run.smithi097:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:15:25.415 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi098.front.sepia.ceph.com 2024-03-25T18:15:25.415 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-25T18:15:25.415 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-25T18:15:25.415 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-25T18:15:25.455 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-25T18:15:25.455 DEBUG:teuthology.orchestra.run.smithi098:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:15:25.509 INFO:tasks.ceph:Setting up mon nodes... 2024-03-25T18:15:25.509 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-25T18:15:25.509 DEBUG:teuthology.orchestra.run.smithi097:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-03-25T18:15:25.563 INFO:teuthology.orchestra.run.smithi097.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-03-25T18:15:25.566 DEBUG:teuthology.orchestra.run.smithi098:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-25T18:15:25.635 INFO:teuthology.orchestra.run.smithi098.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-25T18:15:25.637 INFO:tasks.ceph:Setting up mds nodes... 2024-03-25T18:15:25.637 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-25T18:15:25.638 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-25T18:15:25.680 INFO:teuthology.orchestra.run.smithi097.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-25T18:15:25.695 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-25T18:15:25.736 INFO:teuthology.orchestra.run.smithi098.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-03-25T18:15:25.752 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-25T18:15:25.752 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi097.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi098.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-25T18:15:25.752 DEBUG:teuthology.orchestra.run.smithi097:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-25T18:15:25.778 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T18:15:25.778 INFO:tasks.ceph:role: osd.0 2024-03-25T18:15:25.778 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi097.front.sepia.ceph.com 2024-03-25T18:15:25.778 DEBUG:teuthology.orchestra.run.smithi097:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-25T18:15:25.853 INFO:teuthology.orchestra.run.smithi097.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:15:25.853 INFO:teuthology.orchestra.run.smithi097.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:15:25.853 INFO:teuthology.orchestra.run.smithi097.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:15:25.853 INFO:teuthology.orchestra.run.smithi097.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:15:25.853 INFO:teuthology.orchestra.run.smithi097.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:15:25.854 INFO:teuthology.orchestra.run.smithi097.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:15:25.854 INFO:teuthology.orchestra.run.smithi097.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:15:25.854 INFO:teuthology.orchestra.run.smithi097.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:15:25.854 INFO:teuthology.orchestra.run.smithi097.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:15:25.854 INFO:teuthology.orchestra.run.smithi097.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:15:26.362 INFO:teuthology.orchestra.run.smithi097.stdout:Discarding blocks...Done. 2024-03-25T18:15:26.364 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi097.front.sepia.ceph.com -o noatime 2024-03-25T18:15:26.364 DEBUG:teuthology.orchestra.run.smithi097:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-25T18:15:26.508 DEBUG:teuthology.orchestra.run.smithi097:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-25T18:15:26.544 DEBUG:teuthology.orchestra.run.smithi097:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-25T18:15:26.612 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T18:15:26.613 INFO:tasks.ceph:role: osd.1 2024-03-25T18:15:26.613 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi097.front.sepia.ceph.com 2024-03-25T18:15:26.613 DEBUG:teuthology.orchestra.run.smithi097:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-25T18:15:26.687 INFO:teuthology.orchestra.run.smithi097.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:15:26.687 INFO:teuthology.orchestra.run.smithi097.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:15:26.687 INFO:teuthology.orchestra.run.smithi097.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:15:26.687 INFO:teuthology.orchestra.run.smithi097.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:15:26.688 INFO:teuthology.orchestra.run.smithi097.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:15:26.688 INFO:teuthology.orchestra.run.smithi097.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:15:26.688 INFO:teuthology.orchestra.run.smithi097.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:15:26.688 INFO:teuthology.orchestra.run.smithi097.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:15:26.688 INFO:teuthology.orchestra.run.smithi097.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:15:26.688 INFO:teuthology.orchestra.run.smithi097.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:15:27.140 INFO:teuthology.orchestra.run.smithi097.stdout:Discarding blocks...Done. 2024-03-25T18:15:27.142 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi097.front.sepia.ceph.com -o noatime 2024-03-25T18:15:27.142 DEBUG:teuthology.orchestra.run.smithi097:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-25T18:15:27.234 DEBUG:teuthology.orchestra.run.smithi097:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-25T18:15:27.265 DEBUG:teuthology.orchestra.run.smithi097:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-25T18:15:27.332 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T18:15:27.332 INFO:tasks.ceph:role: osd.2 2024-03-25T18:15:27.332 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi097.front.sepia.ceph.com 2024-03-25T18:15:27.333 DEBUG:teuthology.orchestra.run.smithi097:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-25T18:15:27.406 INFO:teuthology.orchestra.run.smithi097.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:15:27.407 INFO:teuthology.orchestra.run.smithi097.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:15:27.407 INFO:teuthology.orchestra.run.smithi097.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:15:27.407 INFO:teuthology.orchestra.run.smithi097.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:15:27.407 INFO:teuthology.orchestra.run.smithi097.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:15:27.407 INFO:teuthology.orchestra.run.smithi097.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:15:27.407 INFO:teuthology.orchestra.run.smithi097.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:15:27.407 INFO:teuthology.orchestra.run.smithi097.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:15:27.407 INFO:teuthology.orchestra.run.smithi097.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:15:27.407 INFO:teuthology.orchestra.run.smithi097.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:15:27.891 INFO:teuthology.orchestra.run.smithi097.stdout:Discarding blocks...Done. 2024-03-25T18:15:27.893 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi097.front.sepia.ceph.com -o noatime 2024-03-25T18:15:27.893 DEBUG:teuthology.orchestra.run.smithi097:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-25T18:15:28.034 DEBUG:teuthology.orchestra.run.smithi097:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-25T18:15:28.066 DEBUG:teuthology.orchestra.run.smithi097:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-25T18:15:28.135 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T18:15:28.135 INFO:tasks.ceph:role: osd.3 2024-03-25T18:15:28.135 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi097.front.sepia.ceph.com 2024-03-25T18:15:28.135 DEBUG:teuthology.orchestra.run.smithi097:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-25T18:15:28.209 INFO:teuthology.orchestra.run.smithi097.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:15:28.209 INFO:teuthology.orchestra.run.smithi097.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:15:28.210 INFO:teuthology.orchestra.run.smithi097.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:15:28.210 INFO:teuthology.orchestra.run.smithi097.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:15:28.210 INFO:teuthology.orchestra.run.smithi097.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:15:28.210 INFO:teuthology.orchestra.run.smithi097.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:15:28.210 INFO:teuthology.orchestra.run.smithi097.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:15:28.210 INFO:teuthology.orchestra.run.smithi097.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:15:28.210 INFO:teuthology.orchestra.run.smithi097.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:15:28.210 INFO:teuthology.orchestra.run.smithi097.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:15:28.730 INFO:teuthology.orchestra.run.smithi097.stdout:Discarding blocks...Done. 2024-03-25T18:15:28.732 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi097.front.sepia.ceph.com -o noatime 2024-03-25T18:15:28.732 DEBUG:teuthology.orchestra.run.smithi097:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-03-25T18:15:28.878 DEBUG:teuthology.orchestra.run.smithi097:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-25T18:15:28.914 DEBUG:teuthology.orchestra.run.smithi097:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:15:29.014 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:15:29.013+0000 7f950a648640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-03-25T18:15:29.014 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:15:29.013+0000 7f950a648640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-03-25T18:15:29.014 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:15:29.013+0000 7f950a648640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-25T18:15:29.015 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:15:29.013+0000 7f950a648640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-25T18:15:29.015 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:15:29.013+0000 7f950a648640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-25T18:15:29.015 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:15:29.013+0000 7f950a648640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-03-25T18:15:31.865 DEBUG:teuthology.orchestra.run.smithi097:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-25T18:15:31.896 DEBUG:teuthology.orchestra.run.smithi097:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:15:31.989 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:15:31.988+0000 7ff592215640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-03-25T18:15:31.990 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:15:31.988+0000 7ff592215640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-03-25T18:15:31.990 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:15:31.988+0000 7ff592215640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-25T18:15:31.990 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:15:31.988+0000 7ff592215640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-25T18:15:31.990 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:15:31.988+0000 7ff592215640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-25T18:15:31.990 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:15:31.989+0000 7ff592215640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-03-25T18:15:34.832 DEBUG:teuthology.orchestra.run.smithi097:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-25T18:15:34.863 DEBUG:teuthology.orchestra.run.smithi097:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:15:34.955 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:15:34.953+0000 7f4f95919640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-03-25T18:15:34.955 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:15:34.954+0000 7f4f95919640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-03-25T18:15:34.955 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:15:34.954+0000 7f4f95919640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-25T18:15:34.955 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:15:34.954+0000 7f4f95919640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-25T18:15:34.955 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:15:34.954+0000 7f4f95919640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-25T18:15:34.956 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:15:34.954+0000 7f4f95919640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-03-25T18:15:37.804 DEBUG:teuthology.orchestra.run.smithi097:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-25T18:15:37.835 DEBUG:teuthology.orchestra.run.smithi097:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:15:37.927 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:15:37.925+0000 7f47aa611640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-03-25T18:15:37.927 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:15:37.926+0000 7f47aa611640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-03-25T18:15:37.927 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:15:37.926+0000 7f47aa611640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-25T18:15:37.927 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:15:37.926+0000 7f47aa611640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-25T18:15:37.927 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:15:37.926+0000 7f47aa611640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-25T18:15:37.928 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:15:37.926+0000 7f47aa611640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-03-25T18:15:40.768 DEBUG:teuthology.orchestra.run.smithi097:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-25T18:15:40.799 DEBUG:teuthology.orchestra.run.smithi098:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-03-25T18:15:40.830 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T18:15:40.830 INFO:tasks.ceph:role: osd.4 2024-03-25T18:15:40.830 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi098.front.sepia.ceph.com 2024-03-25T18:15:40.831 DEBUG:teuthology.orchestra.run.smithi098:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-25T18:15:40.908 INFO:teuthology.orchestra.run.smithi098.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:15:40.930 INFO:teuthology.orchestra.run.smithi098.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:15:40.930 INFO:teuthology.orchestra.run.smithi098.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:15:40.930 INFO:teuthology.orchestra.run.smithi098.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:15:40.931 INFO:teuthology.orchestra.run.smithi098.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:15:40.931 INFO:teuthology.orchestra.run.smithi098.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:15:40.931 INFO:teuthology.orchestra.run.smithi098.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:15:40.931 INFO:teuthology.orchestra.run.smithi098.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:15:40.931 INFO:teuthology.orchestra.run.smithi098.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:15:40.931 INFO:teuthology.orchestra.run.smithi098.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:15:41.418 INFO:teuthology.orchestra.run.smithi098.stdout:Discarding blocks...Done. 2024-03-25T18:15:41.420 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi098.front.sepia.ceph.com -o noatime 2024-03-25T18:15:41.420 DEBUG:teuthology.orchestra.run.smithi098:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-03-25T18:15:41.582 DEBUG:teuthology.orchestra.run.smithi098:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-03-25T18:15:41.618 DEBUG:teuthology.orchestra.run.smithi098:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-03-25T18:15:41.689 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T18:15:41.689 INFO:tasks.ceph:role: osd.5 2024-03-25T18:15:41.689 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi098.front.sepia.ceph.com 2024-03-25T18:15:41.689 DEBUG:teuthology.orchestra.run.smithi098:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-25T18:15:41.764 INFO:teuthology.orchestra.run.smithi098.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:15:41.764 INFO:teuthology.orchestra.run.smithi098.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:15:41.764 INFO:teuthology.orchestra.run.smithi098.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:15:41.764 INFO:teuthology.orchestra.run.smithi098.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:15:41.764 INFO:teuthology.orchestra.run.smithi098.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:15:41.765 INFO:teuthology.orchestra.run.smithi098.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:15:41.765 INFO:teuthology.orchestra.run.smithi098.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:15:41.765 INFO:teuthology.orchestra.run.smithi098.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:15:41.765 INFO:teuthology.orchestra.run.smithi098.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:15:41.765 INFO:teuthology.orchestra.run.smithi098.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:15:42.208 INFO:teuthology.orchestra.run.smithi098.stdout:Discarding blocks...Done. 2024-03-25T18:15:42.210 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi098.front.sepia.ceph.com -o noatime 2024-03-25T18:15:42.210 DEBUG:teuthology.orchestra.run.smithi098:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-03-25T18:15:42.307 DEBUG:teuthology.orchestra.run.smithi098:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-03-25T18:15:42.340 DEBUG:teuthology.orchestra.run.smithi098:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-03-25T18:15:42.407 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T18:15:42.407 INFO:tasks.ceph:role: osd.6 2024-03-25T18:15:42.407 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi098.front.sepia.ceph.com 2024-03-25T18:15:42.408 DEBUG:teuthology.orchestra.run.smithi098:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-25T18:15:42.482 INFO:teuthology.orchestra.run.smithi098.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:15:42.482 INFO:teuthology.orchestra.run.smithi098.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:15:42.482 INFO:teuthology.orchestra.run.smithi098.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:15:42.482 INFO:teuthology.orchestra.run.smithi098.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:15:42.482 INFO:teuthology.orchestra.run.smithi098.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:15:42.482 INFO:teuthology.orchestra.run.smithi098.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:15:42.482 INFO:teuthology.orchestra.run.smithi098.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:15:42.482 INFO:teuthology.orchestra.run.smithi098.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:15:42.482 INFO:teuthology.orchestra.run.smithi098.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:15:42.483 INFO:teuthology.orchestra.run.smithi098.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:15:42.972 INFO:teuthology.orchestra.run.smithi098.stdout:Discarding blocks...Done. 2024-03-25T18:15:42.974 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi098.front.sepia.ceph.com -o noatime 2024-03-25T18:15:42.974 DEBUG:teuthology.orchestra.run.smithi098:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-03-25T18:15:43.099 DEBUG:teuthology.orchestra.run.smithi098:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-03-25T18:15:43.133 DEBUG:teuthology.orchestra.run.smithi098:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-03-25T18:15:43.199 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T18:15:43.199 INFO:tasks.ceph:role: osd.7 2024-03-25T18:15:43.199 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi098.front.sepia.ceph.com 2024-03-25T18:15:43.199 DEBUG:teuthology.orchestra.run.smithi098:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-25T18:15:43.273 INFO:teuthology.orchestra.run.smithi098.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:15:43.273 INFO:teuthology.orchestra.run.smithi098.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:15:43.273 INFO:teuthology.orchestra.run.smithi098.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:15:43.273 INFO:teuthology.orchestra.run.smithi098.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:15:43.273 INFO:teuthology.orchestra.run.smithi098.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:15:43.273 INFO:teuthology.orchestra.run.smithi098.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:15:43.273 INFO:teuthology.orchestra.run.smithi098.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:15:43.273 INFO:teuthology.orchestra.run.smithi098.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:15:43.274 INFO:teuthology.orchestra.run.smithi098.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:15:43.274 INFO:teuthology.orchestra.run.smithi098.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:15:43.793 INFO:teuthology.orchestra.run.smithi098.stdout:Discarding blocks...Done. 2024-03-25T18:15:43.795 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi098.front.sepia.ceph.com -o noatime 2024-03-25T18:15:43.795 DEBUG:teuthology.orchestra.run.smithi098:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-03-25T18:15:43.927 DEBUG:teuthology.orchestra.run.smithi098:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-03-25T18:15:43.962 DEBUG:teuthology.orchestra.run.smithi098:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:15:44.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:15:44.057+0000 7f3e1bd49640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-03-25T18:15:44.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:15:44.057+0000 7f3e1bd49640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-03-25T18:15:44.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:15:44.057+0000 7f3e1bd49640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-25T18:15:44.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:15:44.057+0000 7f3e1bd49640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-25T18:15:44.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:15:44.058+0000 7f3e1bd49640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-25T18:15:44.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:15:44.058+0000 7f3e1bd49640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-03-25T18:15:46.905 DEBUG:teuthology.orchestra.run.smithi098:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-25T18:15:46.936 DEBUG:teuthology.orchestra.run.smithi098:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:15:47.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:15:47.026+0000 7fe6a3417640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-03-25T18:15:47.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:15:47.026+0000 7fe6a3417640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-03-25T18:15:47.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:15:47.026+0000 7fe6a3417640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-25T18:15:47.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:15:47.026+0000 7fe6a3417640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-25T18:15:47.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:15:47.026+0000 7fe6a3417640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-25T18:15:47.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:15:47.027+0000 7fe6a3417640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-03-25T18:15:49.873 DEBUG:teuthology.orchestra.run.smithi098:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-25T18:15:49.903 DEBUG:teuthology.orchestra.run.smithi098:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:15:49.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:15:49.993+0000 7f706dffa640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-03-25T18:15:49.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:15:49.993+0000 7f706dffa640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-03-25T18:15:49.994 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:15:49.993+0000 7f706dffa640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-25T18:15:49.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:15:49.993+0000 7f706dffa640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-25T18:15:49.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:15:49.993+0000 7f706dffa640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-25T18:15:49.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:15:49.994+0000 7f706dffa640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-03-25T18:15:52.839 DEBUG:teuthology.orchestra.run.smithi098:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-25T18:15:52.870 DEBUG:teuthology.orchestra.run.smithi098:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:15:52.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:15:52.962+0000 7fc30910d640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-03-25T18:15:52.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:15:52.962+0000 7fc30910d640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-03-25T18:15:52.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:15:52.962+0000 7fc30910d640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-25T18:15:52.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:15:52.962+0000 7fc30910d640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-25T18:15:52.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:15:52.962+0000 7fc30910d640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-25T18:15:52.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:15:52.963+0000 7fc30910d640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-03-25T18:15:55.799 DEBUG:teuthology.orchestra.run.smithi098:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-25T18:15:55.830 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-25T18:15:55.830 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-25T18:15:55.830 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-03-25T18:15:55.862 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-25T18:15:55.862 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-25T18:15:55.932 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-25T18:15:55.932 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-25T18:15:56.001 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-25T18:15:56.001 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-25T18:15:56.070 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-25T18:15:56.070 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-25T18:15:56.138 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-25T18:15:56.138 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-25T18:15:56.170 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-25T18:15:56.170 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-03-25T18:15:56.239 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-25T18:15:56.239 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-03-25T18:15:56.308 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-25T18:15:56.308 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-03-25T18:15:56.376 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-25T18:15:56.376 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-03-25T18:15:56.444 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-25T18:15:56.444 DEBUG:teuthology.orchestra.run.smithi097:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-25T18:15:56.460 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-25T18:15:56.461 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-03-25T18:15:56.499 INFO:tasks.ceph:Adding keys to all mons... 2024-03-25T18:15:56.499 DEBUG:teuthology.orchestra.run.smithi097:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-25T18:15:56.503 DEBUG:teuthology.orchestra.run.smithi098:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-25T18:15:56.542 INFO:teuthology.orchestra.run.smithi097.stdout:[mgr.y] 2024-03-25T18:15:56.542 INFO:teuthology.orchestra.run.smithi097.stdout: key = AQA9vwFmoTmbIRAADzymP5ylROWqY7B9P5E5Vg== 2024-03-25T18:15:56.542 INFO:teuthology.orchestra.run.smithi097.stdout:[osd.0] 2024-03-25T18:15:56.542 INFO:teuthology.orchestra.run.smithi097.stdout: key = AQBBvwFmprvbABAAfsrvG2oimMlnEuGdpywBIA== 2024-03-25T18:15:56.542 INFO:teuthology.orchestra.run.smithi097.stdout:[osd.1] 2024-03-25T18:15:56.542 INFO:teuthology.orchestra.run.smithi097.stdout: key = AQBDvwFmaWn6OhAAhJS2nQJBmvL/PU+Rgx0H2A== 2024-03-25T18:15:56.542 INFO:teuthology.orchestra.run.smithi097.stdout:[osd.2] 2024-03-25T18:15:56.543 INFO:teuthology.orchestra.run.smithi097.stdout: key = AQBGvwFmywbrOBAAO9sC9/XjSp61pBS7Ash6RA== 2024-03-25T18:15:56.543 INFO:teuthology.orchestra.run.smithi097.stdout:[osd.3] 2024-03-25T18:15:56.543 INFO:teuthology.orchestra.run.smithi097.stdout: key = AQBJvwFmWmpENxAA4tLY+P3SRdnE/KT/qi7OHA== 2024-03-25T18:15:56.543 INFO:teuthology.orchestra.run.smithi097.stdout:[mgr.x] 2024-03-25T18:15:56.543 INFO:teuthology.orchestra.run.smithi097.stdout: key = AQA9vwFmd53hJRAA9PpVU3YPZXXDZIwmRcuqJg== 2024-03-25T18:15:56.543 INFO:teuthology.orchestra.run.smithi097.stdout:[osd.4] 2024-03-25T18:15:56.543 INFO:teuthology.orchestra.run.smithi097.stdout: key = AQBQvwFmJHp+AxAAzVi5CJeWvPN9IoMWBqfUDg== 2024-03-25T18:15:56.543 INFO:teuthology.orchestra.run.smithi097.stdout:[osd.5] 2024-03-25T18:15:56.543 INFO:teuthology.orchestra.run.smithi097.stdout: key = AQBTvwFmiDOkARAApgzd+EoimhrcuvNs/B5jQg== 2024-03-25T18:15:56.543 INFO:teuthology.orchestra.run.smithi097.stdout:[osd.6] 2024-03-25T18:15:56.543 INFO:teuthology.orchestra.run.smithi097.stdout: key = AQBVvwFmO2BHOxAAv5rVG8AF5RmLEhxvjX+QSg== 2024-03-25T18:15:56.543 INFO:teuthology.orchestra.run.smithi097.stdout:[osd.7] 2024-03-25T18:15:56.543 INFO:teuthology.orchestra.run.smithi097.stdout: key = AQBYvwFmFoJpORAAkkpNJh7kWFxVQd3DwtZjMQ== 2024-03-25T18:15:56.543 INFO:teuthology.orchestra.run.smithi097.stdout:[client.0] 2024-03-25T18:15:56.544 INFO:teuthology.orchestra.run.smithi097.stdout: key = AQA9vwFmA02NKBAAjekcq1FtC3dcNQdFzUZVDA== 2024-03-25T18:15:56.544 INFO:teuthology.orchestra.run.smithi097.stdout:[client.1] 2024-03-25T18:15:56.544 INFO:teuthology.orchestra.run.smithi097.stdout: key = AQA9vwFmZGTxKxAAw0D+17BQMu+YnPPtALMSgA== 2024-03-25T18:15:56.564 INFO:teuthology.orchestra.run.smithi098.stdout:[mgr.y] 2024-03-25T18:15:56.565 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQA9vwFmoTmbIRAADzymP5ylROWqY7B9P5E5Vg== 2024-03-25T18:15:56.565 INFO:teuthology.orchestra.run.smithi098.stdout:[osd.0] 2024-03-25T18:15:56.565 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQBBvwFmprvbABAAfsrvG2oimMlnEuGdpywBIA== 2024-03-25T18:15:56.565 INFO:teuthology.orchestra.run.smithi098.stdout:[osd.1] 2024-03-25T18:15:56.565 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQBDvwFmaWn6OhAAhJS2nQJBmvL/PU+Rgx0H2A== 2024-03-25T18:15:56.565 INFO:teuthology.orchestra.run.smithi098.stdout:[osd.2] 2024-03-25T18:15:56.565 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQBGvwFmywbrOBAAO9sC9/XjSp61pBS7Ash6RA== 2024-03-25T18:15:56.565 INFO:teuthology.orchestra.run.smithi098.stdout:[osd.3] 2024-03-25T18:15:56.565 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQBJvwFmWmpENxAA4tLY+P3SRdnE/KT/qi7OHA== 2024-03-25T18:15:56.565 INFO:teuthology.orchestra.run.smithi098.stdout:[mgr.x] 2024-03-25T18:15:56.565 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQA9vwFmd53hJRAA9PpVU3YPZXXDZIwmRcuqJg== 2024-03-25T18:15:56.565 INFO:teuthology.orchestra.run.smithi098.stdout:[osd.4] 2024-03-25T18:15:56.566 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQBQvwFmJHp+AxAAzVi5CJeWvPN9IoMWBqfUDg== 2024-03-25T18:15:56.566 INFO:teuthology.orchestra.run.smithi098.stdout:[osd.5] 2024-03-25T18:15:56.566 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQBTvwFmiDOkARAApgzd+EoimhrcuvNs/B5jQg== 2024-03-25T18:15:56.566 INFO:teuthology.orchestra.run.smithi098.stdout:[osd.6] 2024-03-25T18:15:56.566 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQBVvwFmO2BHOxAAv5rVG8AF5RmLEhxvjX+QSg== 2024-03-25T18:15:56.566 INFO:teuthology.orchestra.run.smithi098.stdout:[osd.7] 2024-03-25T18:15:56.566 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQBYvwFmFoJpORAAkkpNJh7kWFxVQd3DwtZjMQ== 2024-03-25T18:15:56.566 INFO:teuthology.orchestra.run.smithi098.stdout:[client.0] 2024-03-25T18:15:56.566 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQA9vwFmA02NKBAAjekcq1FtC3dcNQdFzUZVDA== 2024-03-25T18:15:56.566 INFO:teuthology.orchestra.run.smithi098.stdout:[client.1] 2024-03-25T18:15:56.566 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQA9vwFmZGTxKxAAw0D+17BQMu+YnPPtALMSgA== 2024-03-25T18:15:56.567 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T18:15:56.587 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T18:15:56.659 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:15:56.661 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:15:56.713 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:15:56.715 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:15:56.770 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:15:56.772 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:15:56.823 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:15:56.825 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:15:56.876 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T18:15:56.878 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T18:15:56.930 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:15:56.933 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:15:56.984 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:15:56.987 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:15:57.038 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:15:57.040 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:15:57.092 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:15:57.094 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:15:57.146 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T18:15:57.149 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T18:15:57.203 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T18:15:57.205 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T18:15:57.260 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-25T18:15:57.260 DEBUG:teuthology.orchestra.run.smithi097:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-25T18:15:57.287 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-25T18:15:57.383 DEBUG:teuthology.orchestra.run.smithi097:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-25T18:15:57.409 DEBUG:teuthology.orchestra.run.smithi097:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-03-25T18:15:57.476 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-25T18:15:57.572 DEBUG:teuthology.orchestra.run.smithi097:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-03-25T18:15:57.599 DEBUG:teuthology.orchestra.run.smithi098:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-03-25T18:15:57.627 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-25T18:15:57.725 DEBUG:teuthology.orchestra.run.smithi098:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-03-25T18:15:57.752 DEBUG:teuthology.orchestra.run.smithi097:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:15:57.754 DEBUG:teuthology.orchestra.run.smithi098:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:15:57.807 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-25T18:15:57.807 DEBUG:tasks.ceph_manager:running mon.a under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.a.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-25T18:15:57.808 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-25T18:15:57.808 DEBUG:teuthology.orchestra.run.smithi097:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a 2024-03-25T18:15:57.810 INFO:tasks.ceph.mon.a:Started 2024-03-25T18:15:57.810 DEBUG:tasks.ceph_manager:running mon.c under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.c.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-25T18:15:57.810 INFO:tasks.ceph.mon.c:Restarting daemon 2024-03-25T18:15:57.810 DEBUG:teuthology.orchestra.run.smithi097:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c 2024-03-25T18:15:57.812 INFO:tasks.ceph.mon.c:Started 2024-03-25T18:15:57.812 DEBUG:tasks.ceph_manager:running mon.b under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.b.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-25T18:15:57.812 INFO:tasks.ceph.mon.b:Restarting daemon 2024-03-25T18:15:57.812 DEBUG:teuthology.orchestra.run.smithi098:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b 2024-03-25T18:15:57.850 INFO:tasks.ceph.mon.b:Started 2024-03-25T18:15:57.850 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-25T18:15:57.850 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-25T18:15:57.850 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i y 2024-03-25T18:15:57.852 INFO:tasks.ceph.mgr.y:Started 2024-03-25T18:15:57.853 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-25T18:15:57.853 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i x 2024-03-25T18:15:57.855 INFO:tasks.ceph.mgr.x:Started 2024-03-25T18:15:57.855 INFO:tasks.ceph:Setting crush tunables to default 2024-03-25T18:15:57.855 DEBUG:teuthology.orchestra.run.smithi097:> sudo ceph --cluster ceph osd crush tunables default 2024-03-25T18:15:57.954 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:15:57.954+0000 7ff53dcc1640 10 monclient: get_monmap_and_config 2024-03-25T18:15:57.954 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:15:57.954+0000 7ff53dcc1640 10 monclient: build_initial_monmap 2024-03-25T18:15:57.954 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:15:57.954+0000 7ff53dcc1640 10 monclient: monmap: 2024-03-25T18:15:57.954 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 0 2024-03-25T18:15:57.954 INFO:teuthology.orchestra.run.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:15:57.954 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2024-03-25T18:15:57.954452+0000 2024-03-25T18:15:57.954 INFO:teuthology.orchestra.run.smithi097.stderr:created 2024-03-25T18:15:57.954452+0000 2024-03-25T18:15:57.954 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:57.954 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 3 2024-03-25T18:15:57.955 INFO:teuthology.orchestra.run.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:15:57.955 INFO:teuthology.orchestra.run.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T18:15:57.955 INFO:teuthology.orchestra.run.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-b 2024-03-25T18:15:57.955 INFO:teuthology.orchestra.run.smithi097.stderr: 2024-03-25T18:15:57.956 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:15:57.954+0000 7ff53dcc1640 1 Processor -- start 2024-03-25T18:15:57.956 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:15:57.955+0000 7ff53dcc1640 1 -- start start 2024-03-25T18:15:57.956 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:15:57.955+0000 7ff53dcc1640 10 monclient: init 2024-03-25T18:15:57.956 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:15:57.955+0000 7ff53dcc1640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:57.957 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:15:57.955+0000 7ff53dcc1640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:15:57.957 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:15:57.955+0000 7ff53dcc1640 10 monclient(hunting): picked mon.noname-a con 0x7ff5380b4700 addr v1:172.21.15.97:6789/0 2024-03-25T18:15:57.957 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:15:57.955+0000 7ff53dcc1640 10 monclient(hunting): picked mon.noname-b con 0x7ff538142a20 addr v1:172.21.15.97:6790/0 2024-03-25T18:15:57.957 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:15:57.955+0000 7ff53dcc1640 10 monclient(hunting): picked mon.noname-c con 0x7ff5380a96b0 addr v1:172.21.15.98:6789/0 2024-03-25T18:15:57.957 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:15:57.955+0000 7ff53dcc1640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5380a9d60 con 0x7ff5380b4700 2024-03-25T18:15:57.957 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:15:57.955+0000 7ff53dcc1640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff53813ff10 con 0x7ff5380a96b0 2024-03-25T18:15:57.957 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:15:57.955+0000 7ff53dcc1640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5380aa1f0 con 0x7ff538142a20 2024-03-25T18:15:57.957 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:15:57.955+0000 7ff53dcc1640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:57.957 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:15:57.955+0000 7ff53dcc1640 10 monclient(hunting): authenticate will time out at 7570.230596s 2024-03-25T18:15:57.957 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:15:57.955+0000 7ff5377fe640 1 -- >> v1:172.21.15.97:6789/0 conn(0x7ff5380b4700 legacy=0x7ff538142310 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.97:6789/0 2024-03-25T18:15:57.957 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:15:57.955+0000 7ff536ffd640 1 -- >> v1:172.21.15.97:6790/0 conn(0x7ff538142a20 legacy=0x7ff53813f0f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.97:6790/0 2024-03-25T18:15:57.957 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:15:57.955+0000 7ff537fff640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7ff5380a96b0 legacy=0x7ff53813f800 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2024-03-25T18:15:58.156 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:15:58.156+0000 7ff5377fe640 1 -- >> v1:172.21.15.97:6789/0 conn(0x7ff5380b4700 legacy=0x7ff538142310 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.97:6789/0 2024-03-25T18:15:58.156 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:15:58.156+0000 7ff536ffd640 1 -- >> v1:172.21.15.97:6790/0 conn(0x7ff538142a20 legacy=0x7ff53813f0f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.97:6790/0 2024-03-25T18:15:58.156 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:15:58.156+0000 7ff537fff640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7ff5380a96b0 legacy=0x7ff53813f800 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2024-03-25T18:15:58.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:15:58.557+0000 7ff5377fe640 1 -- >> v1:172.21.15.97:6789/0 conn(0x7ff5380b4700 legacy=0x7ff538142310 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.97:6789/0 2024-03-25T18:15:58.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:15:58.557+0000 7ff536ffd640 1 -- >> v1:172.21.15.97:6790/0 conn(0x7ff538142a20 legacy=0x7ff53813f0f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.97:6790/0 2024-03-25T18:15:58.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:15:58.557+0000 7ff537fff640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7ff5380a96b0 legacy=0x7ff53813f800 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2024-03-25T18:15:59.358 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:15:59.358+0000 7ff5377fe640 1 -- >> v1:172.21.15.97:6789/0 conn(0x7ff5380b4700 legacy=0x7ff538142310 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.97:6789/0 2024-03-25T18:15:59.358 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:15:59.358+0000 7ff536ffd640 1 -- >> v1:172.21.15.97:6790/0 conn(0x7ff538142a20 legacy=0x7ff53813f0f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.97:6790/0 2024-03-25T18:15:59.358 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:15:59.358+0000 7ff537fff640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7ff5380a96b0 legacy=0x7ff53813f800 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2024-03-25T18:16:00.956 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:00.954+0000 7ff5357fa640 10 monclient(hunting): tick 2024-03-25T18:16:00.956 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:00.954+0000 7ff5357fa640 10 monclient(hunting): _check_auth_tickets 2024-03-25T18:16:00.956 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:00.954+0000 7ff5357fa640 1 monclient(hunting): continuing hunt 2024-03-25T18:16:00.956 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:00.954+0000 7ff5357fa640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T18:16:00.956 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:00.954+0000 7ff5357fa640 1 -- >> v1:172.21.15.97:6790/0 conn(0x7ff538142a20 legacy=0x7ff53813f0f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:16:00.956 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:00.954+0000 7ff5357fa640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7ff5380a96b0 legacy=0x7ff53813f800 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:16:00.956 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:00.954+0000 7ff5357fa640 1 -- >> v1:172.21.15.97:6789/0 conn(0x7ff5380b4700 legacy=0x7ff538142310 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:16:00.956 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:00.954+0000 7ff5357fa640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:16:00.957 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:00.954+0000 7ff5357fa640 10 monclient(hunting): picked mon.noname-b con 0x7ff538142a20 addr v1:172.21.15.97:6790/0 2024-03-25T18:16:00.957 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:00.954+0000 7ff5357fa640 10 monclient(hunting): picked mon.noname-a con 0x7ff5380b4700 addr v1:172.21.15.97:6789/0 2024-03-25T18:16:00.957 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:00.954+0000 7ff5357fa640 10 monclient(hunting): picked mon.noname-c con 0x7ff5380a96b0 addr v1:172.21.15.98:6789/0 2024-03-25T18:16:00.957 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:00.954+0000 7ff5357fa640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5380b4ae0 con 0x7ff5380b4700 2024-03-25T18:16:00.957 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:00.954+0000 7ff5357fa640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5380a9d60 con 0x7ff5380a96b0 2024-03-25T18:16:00.957 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:00.954+0000 7ff5357fa640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff53813ff10 con 0x7ff538142a20 2024-03-25T18:16:00.957 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:00.954+0000 7ff5357fa640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:00.957 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:00.954+0000 7ff5377fe640 1 -- >> v1:172.21.15.97:6790/0 conn(0x7ff538142a20 legacy=0x7ff5240056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.97:6790/0 2024-03-25T18:16:00.957 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:00.954+0000 7ff536ffd640 1 -- >> v1:172.21.15.97:6789/0 conn(0x7ff5380b4700 legacy=0x7ff524008ed0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.97:6789/0 2024-03-25T18:16:00.957 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:00.954+0000 7ff537fff640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7ff5380a96b0 legacy=0x7ff52400c760 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2024-03-25T18:16:01.156 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:01.155+0000 7ff5377fe640 1 -- >> v1:172.21.15.97:6790/0 conn(0x7ff538142a20 legacy=0x7ff5240056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.97:6790/0 2024-03-25T18:16:01.156 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:01.155+0000 7ff536ffd640 1 -- >> v1:172.21.15.97:6789/0 conn(0x7ff5380b4700 legacy=0x7ff524008ed0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.97:6789/0 2024-03-25T18:16:01.156 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:01.155+0000 7ff537fff640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7ff5380a96b0 legacy=0x7ff52400c760 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2024-03-25T18:16:01.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:01.556+0000 7ff536ffd640 1 -- >> v1:172.21.15.97:6789/0 conn(0x7ff5380b4700 legacy=0x7ff524008ed0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.97:6789/0 2024-03-25T18:16:01.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:01.556+0000 7ff5377fe640 1 -- >> v1:172.21.15.97:6790/0 conn(0x7ff538142a20 legacy=0x7ff5240056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.97:6790/0 2024-03-25T18:16:01.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:01.556+0000 7ff537fff640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7ff5380a96b0 legacy=0x7ff52400c760 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2024-03-25T18:16:02.358 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:02.357+0000 7ff536ffd640 1 -- >> v1:172.21.15.97:6789/0 conn(0x7ff5380b4700 legacy=0x7ff524008ed0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.97:6789/0 2024-03-25T18:16:02.358 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:02.357+0000 7ff5377fe640 1 -- >> v1:172.21.15.97:6790/0 conn(0x7ff538142a20 legacy=0x7ff5240056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.97:6790/0 2024-03-25T18:16:02.358 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:02.357+0000 7ff537fff640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7ff5380a96b0 legacy=0x7ff52400c760 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2024-03-25T18:16:03.956 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:03.954+0000 7ff5357fa640 10 monclient(hunting): tick 2024-03-25T18:16:03.956 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:03.954+0000 7ff5357fa640 10 monclient(hunting): _check_auth_tickets 2024-03-25T18:16:03.957 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:03.954+0000 7ff5357fa640 1 monclient(hunting): continuing hunt 2024-03-25T18:16:03.957 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:03.954+0000 7ff5357fa640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T18:16:03.957 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:03.954+0000 7ff5357fa640 1 -- >> v1:172.21.15.97:6790/0 conn(0x7ff538142a20 legacy=0x7ff5240056d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:16:03.957 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:03.954+0000 7ff5357fa640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7ff5380a96b0 legacy=0x7ff52400c760 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:16:03.957 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:03.954+0000 7ff5357fa640 1 -- >> v1:172.21.15.97:6789/0 conn(0x7ff5380b4700 legacy=0x7ff524008ed0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:16:03.957 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:03.954+0000 7ff5357fa640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:16:03.957 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:03.954+0000 7ff5357fa640 10 monclient(hunting): picked mon.noname-c con 0x7ff52400ffa0 addr v1:172.21.15.98:6789/0 2024-03-25T18:16:03.957 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:03.955+0000 7ff5357fa640 10 monclient(hunting): picked mon.noname-b con 0x7ff538142a20 addr v1:172.21.15.97:6790/0 2024-03-25T18:16:03.957 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:03.955+0000 7ff5357fa640 10 monclient(hunting): picked mon.noname-a con 0x7ff5380b4700 addr v1:172.21.15.97:6789/0 2024-03-25T18:16:03.957 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:03.955+0000 7ff5357fa640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5380aa1f0 con 0x7ff5380b4700 2024-03-25T18:16:03.957 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:03.955+0000 7ff5357fa640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5380b4ae0 con 0x7ff52400ffa0 2024-03-25T18:16:03.957 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:03.955+0000 7ff5357fa640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5380a9d60 con 0x7ff538142a20 2024-03-25T18:16:03.957 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:03.955+0000 7ff5357fa640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:03.957 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:03.955+0000 7ff536ffd640 1 -- >> v1:172.21.15.97:6790/0 conn(0x7ff538142a20 legacy=0x7ff5240056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.97:6790/0 2024-03-25T18:16:03.958 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:03.955+0000 7ff537fff640 1 -- >> v1:172.21.15.97:6789/0 conn(0x7ff5380b4700 legacy=0x7ff524008ed0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.97:6789/0 2024-03-25T18:16:03.958 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:03.955+0000 7ff5377fe640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7ff52400ffa0 legacy=0x7ff524010380 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2024-03-25T18:16:04.157 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:04.156+0000 7ff537fff640 1 -- >> v1:172.21.15.97:6789/0 conn(0x7ff5380b4700 legacy=0x7ff524008ed0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.97:6789/0 2024-03-25T18:16:04.157 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:04.156+0000 7ff536ffd640 1 -- >> v1:172.21.15.97:6790/0 conn(0x7ff538142a20 legacy=0x7ff5240056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.97:6790/0 2024-03-25T18:16:04.157 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:04.156+0000 7ff5377fe640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7ff52400ffa0 legacy=0x7ff524010380 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2024-03-25T18:16:04.558 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:04.557+0000 7ff537fff640 1 -- >> v1:172.21.15.97:6789/0 conn(0x7ff5380b4700 legacy=0x7ff524008ed0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.97:6789/0 2024-03-25T18:16:04.558 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:04.557+0000 7ff536ffd640 1 -- >> v1:172.21.15.97:6790/0 conn(0x7ff538142a20 legacy=0x7ff5240056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.97:6790/0 2024-03-25T18:16:04.558 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:04.557+0000 7ff5377fe640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7ff52400ffa0 legacy=0x7ff524010380 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2024-03-25T18:16:04.841 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:00:06.885 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:04.924 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:00:06.963 34931== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:04.932 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:00:06.971 34935== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:05.149 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:00:07.193 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:05.233 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:00:07.272 34931== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:05.241 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:00:07.280 34935== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:05.359 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:05.358+0000 7ff537fff640 1 -- >> v1:172.21.15.97:6789/0 conn(0x7ff5380b4700 legacy=0x7ff524008ed0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.97:6789/0 2024-03-25T18:16:05.359 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:05.358+0000 7ff536ffd640 1 -- >> v1:172.21.15.97:6790/0 conn(0x7ff538142a20 legacy=0x7ff5240056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.97:6790/0 2024-03-25T18:16:05.359 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:05.358+0000 7ff5377fe640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7ff52400ffa0 legacy=0x7ff524010380 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2024-03-25T18:16:06.956 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:06.955+0000 7ff5357fa640 10 monclient(hunting): tick 2024-03-25T18:16:06.956 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:06.955+0000 7ff5357fa640 10 monclient(hunting): _check_auth_tickets 2024-03-25T18:16:06.957 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:06.955+0000 7ff5357fa640 1 monclient(hunting): continuing hunt 2024-03-25T18:16:06.957 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:06.955+0000 7ff5357fa640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T18:16:06.957 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:06.955+0000 7ff5357fa640 1 -- >> v1:172.21.15.97:6790/0 conn(0x7ff538142a20 legacy=0x7ff5240056d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:16:06.957 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:06.955+0000 7ff5357fa640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7ff52400ffa0 legacy=0x7ff524010380 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:16:06.957 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:06.955+0000 7ff5357fa640 1 -- >> v1:172.21.15.97:6789/0 conn(0x7ff5380b4700 legacy=0x7ff524008ed0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:16:06.957 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:06.955+0000 7ff5357fa640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:16:06.957 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:06.955+0000 7ff5357fa640 10 monclient(hunting): picked mon.noname-c con 0x7ff52400ffa0 addr v1:172.21.15.98:6789/0 2024-03-25T18:16:06.957 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:06.955+0000 7ff5357fa640 10 monclient(hunting): picked mon.noname-b con 0x7ff538142a20 addr v1:172.21.15.97:6790/0 2024-03-25T18:16:06.957 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:06.955+0000 7ff5357fa640 10 monclient(hunting): picked mon.noname-a con 0x7ff5380b4700 addr v1:172.21.15.97:6789/0 2024-03-25T18:16:06.957 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:06.955+0000 7ff5357fa640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff53813ff10 con 0x7ff5380b4700 2024-03-25T18:16:06.957 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:06.955+0000 7ff5357fa640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5380aa1f0 con 0x7ff52400ffa0 2024-03-25T18:16:06.957 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:06.955+0000 7ff5357fa640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5380b4ae0 con 0x7ff538142a20 2024-03-25T18:16:06.957 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:06.955+0000 7ff5357fa640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:06.960 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:06.959+0000 7ff536ffd640 1 --1- >> v1:172.21.15.97:6790/0 conn(0x7ff538142a20 0x7ff5240056d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6790/0 says I am v1:172.21.15.97:45974/0 (socket says 172.21.15.97:45974) 2024-03-25T18:16:06.960 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:06.959+0000 7ff536ffd640 1 -- 172.21.15.97:0/2490873284 learned_addr learned my addr 172.21.15.97:0/2490873284 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:16:09.956 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:09.955+0000 7ff5357fa640 10 monclient(hunting): tick 2024-03-25T18:16:09.957 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:09.955+0000 7ff5357fa640 10 monclient(hunting): _check_auth_tickets 2024-03-25T18:16:09.957 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:09.955+0000 7ff5357fa640 1 monclient(hunting): continuing hunt 2024-03-25T18:16:09.957 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:09.955+0000 7ff5357fa640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T18:16:09.957 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:09.955+0000 7ff5357fa640 1 -- 172.21.15.97:0/2490873284 >> v1:172.21.15.97:6790/0 conn(0x7ff538142a20 legacy=0x7ff5240056d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:09.957 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:09.955+0000 7ff5357fa640 1 -- 172.21.15.97:0/2490873284 >> v1:172.21.15.98:6789/0 conn(0x7ff52400ffa0 legacy=0x7ff524010380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:09.957 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:09.955+0000 7ff5357fa640 1 -- 172.21.15.97:0/2490873284 >> v1:172.21.15.97:6789/0 conn(0x7ff5380b4700 legacy=0x7ff524008ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:09.957 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:09.955+0000 7ff5357fa640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:16:09.957 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:09.955+0000 7ff5357fa640 10 monclient(hunting): picked mon.noname-b con 0x7ff52400a750 addr v1:172.21.15.97:6790/0 2024-03-25T18:16:09.957 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:09.955+0000 7ff5357fa640 10 monclient(hunting): picked mon.noname-c con 0x7ff52400ffa0 addr v1:172.21.15.98:6789/0 2024-03-25T18:16:09.957 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:09.955+0000 7ff5357fa640 10 monclient(hunting): picked mon.noname-a con 0x7ff5380b4700 addr v1:172.21.15.97:6789/0 2024-03-25T18:16:09.957 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:09.955+0000 7ff5357fa640 1 -- 172.21.15.97:0/2490873284 --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5380a9d60 con 0x7ff5380b4700 2024-03-25T18:16:09.957 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:09.955+0000 7ff5357fa640 1 -- 172.21.15.97:0/2490873284 --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff52400dcd0 con 0x7ff52400ffa0 2024-03-25T18:16:09.957 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:09.955+0000 7ff5357fa640 1 -- 172.21.15.97:0/2490873284 --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff524010d40 con 0x7ff52400a750 2024-03-25T18:16:09.957 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:09.955+0000 7ff5357fa640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:12.957 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:12.955+0000 7ff5357fa640 10 monclient(hunting): tick 2024-03-25T18:16:12.957 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:12.955+0000 7ff5357fa640 10 monclient(hunting): _check_auth_tickets 2024-03-25T18:16:12.957 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:12.955+0000 7ff5357fa640 1 monclient(hunting): continuing hunt 2024-03-25T18:16:12.957 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:12.955+0000 7ff5357fa640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T18:16:12.957 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:12.955+0000 7ff5357fa640 1 -- 172.21.15.97:0/2490873284 >> v1:172.21.15.97:6790/0 conn(0x7ff52400a750 legacy=0x7ff524011060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:12.957 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:12.955+0000 7ff5357fa640 1 -- 172.21.15.97:0/2490873284 >> v1:172.21.15.98:6789/0 conn(0x7ff52400ffa0 legacy=0x7ff524010380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:12.958 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:12.955+0000 7ff5357fa640 1 -- 172.21.15.97:0/2490873284 >> v1:172.21.15.97:6789/0 conn(0x7ff5380b4700 legacy=0x7ff524008eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:12.958 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:12.955+0000 7ff5357fa640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:16:12.958 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:12.955+0000 7ff5357fa640 10 monclient(hunting): picked mon.noname-a con 0x7ff524005870 addr v1:172.21.15.97:6789/0 2024-03-25T18:16:12.958 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:12.955+0000 7ff5357fa640 10 monclient(hunting): picked mon.noname-b con 0x7ff52400a750 addr v1:172.21.15.97:6790/0 2024-03-25T18:16:12.958 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:12.955+0000 7ff5357fa640 10 monclient(hunting): picked mon.noname-c con 0x7ff52400ffa0 addr v1:172.21.15.98:6789/0 2024-03-25T18:16:12.958 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:12.955+0000 7ff5357fa640 1 -- 172.21.15.97:0/2490873284 --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5240056c0 con 0x7ff524005870 2024-03-25T18:16:12.958 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:12.955+0000 7ff5357fa640 1 -- 172.21.15.97:0/2490873284 --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff524012070 con 0x7ff52400ffa0 2024-03-25T18:16:12.958 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:12.955+0000 7ff5357fa640 1 -- 172.21.15.97:0/2490873284 --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff524012200 con 0x7ff52400a750 2024-03-25T18:16:12.958 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:12.955+0000 7ff5357fa640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:15.786 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:15.785+0000 7ff5367fc640 1 -- 172.21.15.97:0/2490873284 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2364320302 0 0) 0x7ff5240056c0 con 0x7ff524005870 2024-03-25T18:16:15.786 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:15.785+0000 7ff5367fc640 10 monclient(hunting): _negotiate 2024-03-25T18:16:15.786 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:15.785+0000 7ff5367fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:15.786 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:15.785+0000 7ff5367fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:15.786 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:15.785+0000 7ff5367fc640 10 monclient(hunting): my global_id is 4101 2024-03-25T18:16:15.786 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:15.785+0000 7ff5367fc640 1 -- 172.21.15.97:0/2490873284 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff520003520 con 0x7ff524005870 2024-03-25T18:16:15.824 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:15.823+0000 7ff5367fc640 1 -- 172.21.15.97:0/2490873284 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2567194538 0 0) 0x7ff524012200 con 0x7ff52400a750 2024-03-25T18:16:15.824 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:15.823+0000 7ff5367fc640 10 monclient(hunting): _negotiate 2024-03-25T18:16:15.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:15.823+0000 7ff5367fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:15.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:15.823+0000 7ff5367fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:15.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:15.823+0000 7ff5367fc640 10 monclient(hunting): my global_id is 4106 2024-03-25T18:16:15.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:15.823+0000 7ff5367fc640 1 -- 172.21.15.97:0/2490873284 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5240056c0 con 0x7ff52400a750 2024-03-25T18:16:15.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:15.824+0000 7ff5367fc640 1 -- 172.21.15.97:0/2490873284 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 953519885 0 0) 0x7ff524012070 con 0x7ff52400ffa0 2024-03-25T18:16:15.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:15.824+0000 7ff5367fc640 10 monclient(hunting): _negotiate 2024-03-25T18:16:15.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:15.824+0000 7ff5367fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:15.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:15.824+0000 7ff5367fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:15.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:15.824+0000 7ff5367fc640 10 monclient(hunting): my global_id is 4105 2024-03-25T18:16:15.826 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:15.824+0000 7ff5367fc640 1 -- 172.21.15.97:0/2490873284 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff524012200 con 0x7ff52400ffa0 2024-03-25T18:16:15.846 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:15.845+0000 7ff5367fc640 1 -- 172.21.15.97:0/2490873284 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3929074065 0 0) 0x7ff520003520 con 0x7ff524005870 2024-03-25T18:16:15.846 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:15.845+0000 7ff5367fc640 1 -- 172.21.15.97:0/2490873284 >> v1:172.21.15.97:6790/0 conn(0x7ff52400a750 legacy=0x7ff52400d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:15.846 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:15.845+0000 7ff5367fc640 1 -- 172.21.15.97:0/2490873284 >> v1:172.21.15.98:6789/0 conn(0x7ff52400ffa0 legacy=0x7ff524010380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:15.846 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:15.845+0000 7ff5367fc640 10 monclient: _finish_hunting 0 2024-03-25T18:16:15.847 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:15.845+0000 7ff5367fc640 1 monclient: found mon.noname-a 2024-03-25T18:16:15.847 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:15.845+0000 7ff5367fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:15.847 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:15.845+0000 7ff5367fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.97:6789/0 2024-03-25T18:16:15.847 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:15.845+0000 7ff5367fc640 1 -- 172.21.15.97:0/2490873284 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff524012390 con 0x7ff524005870 2024-03-25T18:16:15.847 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:15.845+0000 7ff5367fc640 10 monclient: _finish_auth 0 2024-03-25T18:16:15.847 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:15.845+0000 7ff5367fc640 10 monclient: _check_auth_tickets 2024-03-25T18:16:15.847 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:15.845+0000 7ff5367fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:15.847 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:15.845+0000 7ff5367fc640 1 -- 172.21.15.97:0/2490873284 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7ff52c004150 con 0x7ff524005870 2024-03-25T18:16:15.847 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:15.845+0000 7ff5367fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:15.847 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:15.845+0000 7ff5367fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:16:15.847 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:15.845+0000 7ff5367fc640 10 monclient: dump: 2024-03-25T18:16:15.847 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2024-03-25T18:16:15.847 INFO:teuthology.orchestra.run.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:16:15.847 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:15.847 INFO:teuthology.orchestra.run.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:15.847 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:15.848 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 3 2024-03-25T18:16:15.848 INFO:teuthology.orchestra.run.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:16:15.848 INFO:teuthology.orchestra.run.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:16:15.848 INFO:teuthology.orchestra.run.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:16:15.848 INFO:teuthology.orchestra.run.smithi097.stderr: 2024-03-25T18:16:15.848 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:15.845+0000 7ff53dcc1640 5 monclient: authenticate success, global_id 4101 2024-03-25T18:16:15.848 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:15.845+0000 7ff53dcc1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:15.924 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:15.923+0000 7ff5367fc640 1 -- 172.21.15.97:0/2490873284 <== mon.0 v1:172.21.15.97:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff52c004320 con 0x7ff524005870 2024-03-25T18:16:15.924 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:15.923+0000 7ff5367fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:15.924 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:15.923+0000 7ff5367fc640 1 -- 172.21.15.97:0/2490873284 <== mon.0 v1:172.21.15.97:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7ff52c004730 con 0x7ff524005870 2024-03-25T18:16:15.924 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:15.923+0000 7ff5367fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:15.924 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:15.923+0000 7ff5367fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:16:15.925 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:15.923+0000 7ff5367fc640 10 monclient: dump: 2024-03-25T18:16:15.925 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2024-03-25T18:16:15.925 INFO:teuthology.orchestra.run.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:16:15.925 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:15.925 INFO:teuthology.orchestra.run.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:15.925 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:15.925 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 3 2024-03-25T18:16:15.925 INFO:teuthology.orchestra.run.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:16:15.925 INFO:teuthology.orchestra.run.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:16:15.925 INFO:teuthology.orchestra.run.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:16:15.925 INFO:teuthology.orchestra.run.smithi097.stderr: 2024-03-25T18:16:15.925 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:15.923+0000 7ff53dcc1640 10 monclient: get_monmap_and_config success 2024-03-25T18:16:15.926 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:15.923+0000 7ff53dcc1640 4 set_mon_vals no callback set 2024-03-25T18:16:15.928 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:15.924+0000 7ff53dcc1640 10 monclient: shutdown 2024-03-25T18:16:15.928 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:15.924+0000 7ff53dcc1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:15.928 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:15.924+0000 7ff53dcc1640 1 -- 172.21.15.97:0/2490873284 >> v1:172.21.15.97:6789/0 conn(0x7ff524005870 legacy=0x7ff52400cc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:15.928 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:15.924+0000 7ff53dcc1640 1 -- 172.21.15.97:0/2490873284 shutdown_connections 2024-03-25T18:16:15.928 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:15.924+0000 7ff53dcc1640 1 -- 172.21.15.97:0/2490873284 >> 172.21.15.97:0/2490873284 conn(0x7ff5380668e0 msgr2=0x7ff538066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:15.928 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:15.924+0000 7ff53dcc1640 1 -- 172.21.15.97:0/2490873284 shutdown_connections 2024-03-25T18:16:15.928 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:15.924+0000 7ff53dcc1640 1 -- 172.21.15.97:0/2490873284 wait complete. 2024-03-25T18:16:15.928 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:15.924+0000 7ff53dcc1640 10 monclient: build_initial_monmap 2024-03-25T18:16:15.928 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:15.924+0000 7ff53dcc1640 10 monclient: monmap: 2024-03-25T18:16:15.929 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 0 2024-03-25T18:16:15.929 INFO:teuthology.orchestra.run.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:16:15.929 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 0.000000 2024-03-25T18:16:15.929 INFO:teuthology.orchestra.run.smithi097.stderr:created 0.000000 2024-03-25T18:16:15.929 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:15.929 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2024-03-25T18:16:15.929 INFO:teuthology.orchestra.run.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:16:15.929 INFO:teuthology.orchestra.run.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T18:16:15.929 INFO:teuthology.orchestra.run.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-c 2024-03-25T18:16:15.929 INFO:teuthology.orchestra.run.smithi097.stderr: 2024-03-25T18:16:15.929 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:15.925+0000 7ff53dcc1640 1 librados: starting msgr at 2024-03-25T18:16:15.929 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:15.925+0000 7ff53dcc1640 1 librados: starting objecter 2024-03-25T18:16:15.929 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:15.925+0000 7ff53dcc1640 1 Processor -- start 2024-03-25T18:16:15.929 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:15.925+0000 7ff53dcc1640 1 -- start start 2024-03-25T18:16:15.929 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:15.925+0000 7ff53dcc1640 1 librados: setting wanted keys 2024-03-25T18:16:15.929 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:15.925+0000 7ff53dcc1640 1 librados: calling monclient init 2024-03-25T18:16:15.930 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:15.925+0000 7ff53dcc1640 10 monclient: init 2024-03-25T18:16:15.930 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:15.926+0000 7ff53dcc1640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:15.930 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:15.926+0000 7ff53dcc1640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:16:15.930 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:15.926+0000 7ff53dcc1640 10 monclient(hunting): picked mon.noname-a con 0x7ff524005870 addr v1:172.21.15.97:6789/0 2024-03-25T18:16:15.930 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:15.926+0000 7ff53dcc1640 10 monclient(hunting): picked mon.noname-b con 0x7ff52400a750 addr v1:172.21.15.98:6789/0 2024-03-25T18:16:15.930 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:15.926+0000 7ff53dcc1640 10 monclient(hunting): picked mon.noname-c con 0x7ff52400ffa0 addr v1:172.21.15.97:6790/0 2024-03-25T18:16:15.930 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:15.926+0000 7ff53dcc1640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5381dba40 con 0x7ff524005870 2024-03-25T18:16:15.930 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:15.926+0000 7ff53dcc1640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5381db710 con 0x7ff52400a750 2024-03-25T18:16:15.930 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:15.926+0000 7ff53dcc1640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff53811fa40 con 0x7ff52400ffa0 2024-03-25T18:16:15.930 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:15.926+0000 7ff53dcc1640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:15.930 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:15.926+0000 7ff53dcc1640 10 monclient(hunting): authenticate will time out at 7588.202646s 2024-03-25T18:16:15.930 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:15.927+0000 7ff537fff640 1 --1- >> v1:172.21.15.97:6790/0 conn(0x7ff52400ffa0 0x7ff5380b0a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.97:6790/0 says I am v1:172.21.15.97:52802/0 (socket says 172.21.15.97:52802) 2024-03-25T18:16:15.930 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:15.927+0000 7ff537fff640 1 -- 172.21.15.97:0/3753301658 learned_addr learned my addr 172.21.15.97:0/3753301658 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:16:15.931 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:15.927+0000 7ff51ffff640 10 client.?.objecter ms_handle_connect 0x7ff52400ffa0 2024-03-25T18:16:15.931 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:15.927+0000 7ff51ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:15.934 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:15.933+0000 7ff51ffff640 10 client.?.objecter ms_handle_connect 0x7ff52400a750 2024-03-25T18:16:15.934 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:15.933+0000 7ff51ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:15.936 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:15.935+0000 7ff51ffff640 10 client.?.objecter ms_handle_connect 0x7ff524005870 2024-03-25T18:16:15.936 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:15.935+0000 7ff51ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:15.942 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:15.941+0000 7ff51ffff640 1 -- 172.21.15.97:0/3753301658 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3452074545 0 0) 0x7ff5381dba40 con 0x7ff524005870 2024-03-25T18:16:15.942 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:15.941+0000 7ff51ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:16:15.942 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:15.941+0000 7ff51ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:15.942 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:15.941+0000 7ff51ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:15.942 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:15.941+0000 7ff51ffff640 10 monclient(hunting): my global_id is 4107 2024-03-25T18:16:15.942 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:15.941+0000 7ff51ffff640 1 -- 172.21.15.97:0/3753301658 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff524010460 con 0x7ff524005870 2024-03-25T18:16:15.978 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:15.977+0000 7ff51ffff640 1 -- 172.21.15.97:0/3753301658 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 867377586 0 0) 0x7ff524010460 con 0x7ff524005870 2024-03-25T18:16:15.978 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:15.977+0000 7ff51ffff640 1 -- 172.21.15.97:0/3753301658 --> v1:172.21.15.97:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff5381dba40 con 0x7ff524005870 2024-03-25T18:16:15.978 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:15.977+0000 7ff51ffff640 1 -- 172.21.15.97:0/3753301658 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7ff52c004630 con 0x7ff524005870 2024-03-25T18:16:15.978 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:15.977+0000 7ff51ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:15.978 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:15.977+0000 7ff51ffff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:16:15.978 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:15.977+0000 7ff51ffff640 10 monclient(hunting): dump: 2024-03-25T18:16:15.978 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2024-03-25T18:16:15.978 INFO:teuthology.orchestra.run.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:16:15.978 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:15.979 INFO:teuthology.orchestra.run.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:15.979 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:15.979 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 3 2024-03-25T18:16:15.979 INFO:teuthology.orchestra.run.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:16:15.979 INFO:teuthology.orchestra.run.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:16:15.979 INFO:teuthology.orchestra.run.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:16:15.979 INFO:teuthology.orchestra.run.smithi097.stderr: 2024-03-25T18:16:15.979 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:15.977+0000 7ff51ffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:16:15.999 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:15.996+0000 7ff51ffff640 1 -- 172.21.15.97:0/3753301658 <== mon.0 v1:172.21.15.97:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4259190486 0 0) 0x7ff5381dba40 con 0x7ff524005870 2024-03-25T18:16:15.999 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:15.996+0000 7ff51ffff640 1 -- 172.21.15.97:0/3753301658 >> v1:172.21.15.97:6790/0 conn(0x7ff52400ffa0 legacy=0x7ff5380b0a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:15.999 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:15.997+0000 7ff51ffff640 1 -- 172.21.15.97:0/3753301658 >> v1:172.21.15.98:6789/0 conn(0x7ff52400a750 legacy=0x7ff5380b0320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:15.999 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:15.997+0000 7ff51ffff640 10 monclient: _finish_hunting 0 2024-03-25T18:16:15.999 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:15.997+0000 7ff51ffff640 1 monclient: found mon.a 2024-03-25T18:16:15.999 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:15.997+0000 7ff51ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:15.999 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:15.997+0000 7ff51ffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:16:15.999 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:15.997+0000 7ff51ffff640 1 -- 172.21.15.97:0/3753301658 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff53811fbd0 con 0x7ff524005870 2024-03-25T18:16:15.999 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:15.997+0000 7ff51ffff640 10 monclient: _finish_auth 0 2024-03-25T18:16:15.999 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:15.997+0000 7ff51ffff640 10 monclient: _check_auth_tickets 2024-03-25T18:16:15.999 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:15.997+0000 7ff51ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:15.999 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:15.997+0000 7ff53dcc1640 5 monclient: authenticate success, global_id 4107 2024-03-25T18:16:16.000 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:15.997+0000 7ff53dcc1640 10 monclient: _renew_subs 2024-03-25T18:16:16.000 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:15.997+0000 7ff53dcc1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:16:16.000 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:15.997+0000 7ff53dcc1640 1 -- 172.21.15.97:0/3753301658 --> v1:172.21.15.97:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff53813a360 con 0x7ff524005870 2024-03-25T18:16:16.000 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:15.997+0000 7ff53dcc1640 10 client.4107.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:16:16.000 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:15.997+0000 7ff53dcc1640 10 monclient: _renew_subs 2024-03-25T18:16:16.000 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:15.997+0000 7ff53dcc1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:16:16.000 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:15.997+0000 7ff53dcc1640 1 -- 172.21.15.97:0/3753301658 --> v1:172.21.15.97:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5381e3350 con 0x7ff524005870 2024-03-25T18:16:16.000 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:15.997+0000 7ff53dcc1640 1 librados: init done 2024-03-25T18:16:16.000 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:15.998+0000 7ff53dcc1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:16.000 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:15.998+0000 7ff53dcc1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:16.000 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:15.998+0000 7ff53dcc1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:16:16.000 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:15.998+0000 7ff53dcc1640 1 -- 172.21.15.97:0/3753301658 --> v1:172.21.15.97:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff53804f730 con 0x7ff524005870 2024-03-25T18:16:16.211 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:16.210+0000 7ff51ffff640 1 -- 172.21.15.97:0/3753301658 <== mon.0 v1:172.21.15.97:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff52c004a40 con 0x7ff524005870 2024-03-25T18:16:16.211 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:16.210+0000 7ff51ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:16.211 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:16.210+0000 7ff51ffff640 1 -- 172.21.15.97:0/3753301658 <== mon.0 v1:172.21.15.97:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7ff52c004e50 con 0x7ff524005870 2024-03-25T18:16:16.211 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:16.210+0000 7ff51ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:16.212 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:16.210+0000 7ff51ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:16:16.212 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:16.210+0000 7ff51ffff640 10 monclient: dump: 2024-03-25T18:16:16.212 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2024-03-25T18:16:16.212 INFO:teuthology.orchestra.run.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:16:16.212 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:16.212 INFO:teuthology.orchestra.run.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:16.212 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:16.212 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 3 2024-03-25T18:16:16.212 INFO:teuthology.orchestra.run.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:16:16.212 INFO:teuthology.orchestra.run.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:16:16.212 INFO:teuthology.orchestra.run.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:16:16.212 INFO:teuthology.orchestra.run.smithi097.stderr: 2024-03-25T18:16:16.212 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:16.210+0000 7ff51ffff640 1 -- 172.21.15.97:0/3753301658 <== mon.0 v1:172.21.15.97:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7ff52c005400 con 0x7ff524005870 2024-03-25T18:16:16.213 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:16.210+0000 7ff535ffb640 4 set_mon_vals no callback set 2024-03-25T18:16:16.213 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:16.210+0000 7ff51ffff640 1 -- 172.21.15.97:0/3753301658 <== mon.0 v1:172.21.15.97:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (unknown 2028182594 0 0) 0x7ff52c006890 con 0x7ff524005870 2024-03-25T18:16:16.213 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:16.210+0000 7ff51ffff640 10 client.4107.objecter ms_dispatch 0x7ff538000b90 osd_map(1..1 src has 1..1) v4 2024-03-25T18:16:16.213 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:16.210+0000 7ff51ffff640 3 client.4107.objecter handle_osd_map got epochs [1,1] > 0 2024-03-25T18:16:16.213 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:16.210+0000 7ff51ffff640 3 client.4107.objecter handle_osd_map decoding full epoch 1 2024-03-25T18:16:16.213 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:16.210+0000 7ff51ffff640 20 client.4107.objecter dump_active .. 0 homeless 2024-03-25T18:16:16.213 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:16.210+0000 7ff51ffff640 1 -- 172.21.15.97:0/3753301658 <== mon.0 v1:172.21.15.97:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7ff52c006b00 con 0x7ff524005870 2024-03-25T18:16:16.213 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:16.210+0000 7ff51ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:16.213 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:16.210+0000 7ff51ffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:16:16.301 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:16.300+0000 7ff53dcc1640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-25T18:16:16.301 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:16.300+0000 7ff53dcc1640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-25T18:16:16.302 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:16.300+0000 7ff53dcc1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:16:16.302 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:16.300+0000 7ff53dcc1640 1 -- 172.21.15.97:0/3753301658 --> v1:172.21.15.97:6789/0 -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7ff5380b17e0 con 0x7ff524005870 2024-03-25T18:16:16.526 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:16.525+0000 7ff51ffff640 1 -- 172.21.15.97:0/3753301658 <== mon.0 v1:172.21.15.97:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (unknown 2277712955 0 0) 0x7ff52c005970 con 0x7ff524005870 2024-03-25T18:16:16.526 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:16.525+0000 7ff51ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-25T18:16:16.526 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:16.525+0000 7ff51ffff640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-03-25T18:16:16.527 INFO:teuthology.orchestra.run.smithi097.stderr:adjusted tunables profile to default 2024-03-25T18:16:16.528 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:16.526+0000 7ff53dcc1640 10 librados: watch_flush enter 2024-03-25T18:16:16.528 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:16.526+0000 7ff53dcc1640 10 librados: watch_flush exit 2024-03-25T18:16:16.528 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:16.526+0000 7ff53dcc1640 20 client.4107.objecter shutdown clearing up homeless session... 2024-03-25T18:16:16.528 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:16.526+0000 7ff53dcc1640 10 client.4107.objecter successfully canceled tick 2024-03-25T18:16:16.528 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:16.526+0000 7ff53dcc1640 10 monclient: shutdown 2024-03-25T18:16:16.528 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:16.526+0000 7ff53dcc1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:16.528 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:16.526+0000 7ff53dcc1640 1 -- 172.21.15.97:0/3753301658 >> v1:172.21.15.97:6789/0 conn(0x7ff524005870 legacy=0x7ff5381db000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:16.528 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:16.526+0000 7ff53dcc1640 1 -- 172.21.15.97:0/3753301658 shutdown_connections 2024-03-25T18:16:16.528 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:16.526+0000 7ff53dcc1640 1 -- 172.21.15.97:0/3753301658 >> 172.21.15.97:0/3753301658 conn(0x7ff5380668e0 msgr2=0x7ff5380b2510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:16.528 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:16.526+0000 7ff53dcc1640 1 -- 172.21.15.97:0/3753301658 shutdown_connections 2024-03-25T18:16:16.529 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:16.526+0000 7ff53dcc1640 1 -- 172.21.15.97:0/3753301658 wait complete. 2024-03-25T18:16:16.529 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:16.526+0000 7ff53dcc1640 1 librados: shutdown 2024-03-25T18:16:16.538 INFO:tasks.ceph:check_enable_crimson: False 2024-03-25T18:16:16.538 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-25T18:16:16.538 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-25T18:16:16.538 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-25T18:16:16.565 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-25T18:16:16.565 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-25T18:16:16.632 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-25T18:16:16.632 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-25T18:16:16.700 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-25T18:16:16.700 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-25T18:16:16.767 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-25T18:16:16.768 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-03-25T18:16:16.774 INFO:tasks.ceph.mgr.x.smithi098.stderr:2024-03-25T18:16:16.773+0000 7f16671db040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-25T18:16:16.796 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-25T18:16:16.796 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-03-25T18:16:16.831 INFO:tasks.ceph.mgr.y.smithi097.stderr:2024-03-25T18:16:16.830+0000 7f7cdf810040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-25T18:16:16.832 INFO:tasks.ceph.mgr.x.smithi098.stderr:2024-03-25T18:16:16.831+0000 7f16671db040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-25T18:16:16.858 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-25T18:16:16.858 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-03-25T18:16:16.888 INFO:tasks.ceph.mgr.y.smithi097.stderr:2024-03-25T18:16:16.887+0000 7f7cdf810040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-25T18:16:16.925 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-25T18:16:16.925 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-03-25T18:16:16.992 DEBUG:teuthology.orchestra.run.smithi098:> sudo ceph --cluster ceph osd new cb5152b9-262f-45c3-94b8-18d0e81df558 0 2024-03-25T18:16:17.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.131+0000 7f59d1ca8640 10 monclient: get_monmap_and_config 2024-03-25T18:16:17.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.131+0000 7f59d1ca8640 10 monclient: build_initial_monmap 2024-03-25T18:16:17.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.131+0000 7f59d1ca8640 10 monclient: monmap: 2024-03-25T18:16:17.132 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-03-25T18:16:17.132 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:16:17.132 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-03-25T18:16:17.132179+0000 2024-03-25T18:16:17.132 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-03-25T18:16:17.132179+0000 2024-03-25T18:16:17.132 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:17.132 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-03-25T18:16:17.132 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:16:17.132 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T18:16:17.132 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.97:6790/0 mon.noname-b 2024-03-25T18:16:17.132 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-25T18:16:17.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.131+0000 7f59d1ca8640 1 Processor -- start 2024-03-25T18:16:17.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.131+0000 7f59d1ca8640 1 -- start start 2024-03-25T18:16:17.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.131+0000 7f59d1ca8640 10 monclient: init 2024-03-25T18:16:17.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.132+0000 7f59d1ca8640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:17.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.132+0000 7f59d1ca8640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:16:17.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.132+0000 7f59d1ca8640 10 monclient(hunting): picked mon.noname-c con 0x7f59cc0b4700 addr v1:172.21.15.98:6789/0 2024-03-25T18:16:17.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.132+0000 7f59d1ca8640 10 monclient(hunting): picked mon.noname-a con 0x7f59cc142a20 addr v1:172.21.15.97:6789/0 2024-03-25T18:16:17.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.132+0000 7f59d1ca8640 10 monclient(hunting): picked mon.noname-b con 0x7f59cc0a96b0 addr v1:172.21.15.97:6790/0 2024-03-25T18:16:17.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.132+0000 7f59d1ca8640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59cc0a9d60 con 0x7f59cc142a20 2024-03-25T18:16:17.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.132+0000 7f59d1ca8640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59cc13ff10 con 0x7f59cc0b4700 2024-03-25T18:16:17.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.132+0000 7f59d1ca8640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59cc0aa1f0 con 0x7f59cc0a96b0 2024-03-25T18:16:17.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.132+0000 7f59d1ca8640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:17.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.132+0000 7f59d1ca8640 10 monclient(hunting): authenticate will time out at 7577.270495s 2024-03-25T18:16:17.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.133+0000 7f59cb7fe640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f59cc0b4700 0x7f59cc142310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:60620/0 (socket says 172.21.15.98:60620) 2024-03-25T18:16:17.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.133+0000 7f59cb7fe640 1 -- 172.21.15.98:0/1466217365 learned_addr learned my addr 172.21.15.98:0/1466217365 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-25T18:16:17.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.135+0000 7f59ca7fc640 1 -- 172.21.15.98:0/1466217365 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1807394320 0 0) 0x7f59cc0aa1f0 con 0x7f59cc0a96b0 2024-03-25T18:16:17.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.135+0000 7f59ca7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:16:17.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.135+0000 7f59ca7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:17.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.135+0000 7f59ca7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:17.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.135+0000 7f59ca7fc640 10 monclient(hunting): my global_id is 4109 2024-03-25T18:16:17.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.135+0000 7f59ca7fc640 1 -- 172.21.15.98:0/1466217365 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59b4003520 con 0x7f59cc0a96b0 2024-03-25T18:16:17.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.135+0000 7f59ca7fc640 1 -- 172.21.15.98:0/1466217365 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1717504859 0 0) 0x7f59cc13ff10 con 0x7f59cc0b4700 2024-03-25T18:16:17.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.135+0000 7f59ca7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:16:17.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.135+0000 7f59ca7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:17.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.135+0000 7f59ca7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:17.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.135+0000 7f59ca7fc640 10 monclient(hunting): my global_id is 4108 2024-03-25T18:16:17.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.135+0000 7f59ca7fc640 1 -- 172.21.15.98:0/1466217365 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59cc0aa1f0 con 0x7f59cc0b4700 2024-03-25T18:16:17.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.136+0000 7f59ca7fc640 1 -- 172.21.15.98:0/1466217365 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1983124778 0 0) 0x7f59cc0a9d60 con 0x7f59cc142a20 2024-03-25T18:16:17.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.136+0000 7f59ca7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:16:17.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.136+0000 7f59ca7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:17.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.136+0000 7f59ca7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:17.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.136+0000 7f59ca7fc640 10 monclient(hunting): my global_id is 4116 2024-03-25T18:16:17.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.136+0000 7f59ca7fc640 1 -- 172.21.15.98:0/1466217365 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59cc13ff10 con 0x7f59cc142a20 2024-03-25T18:16:17.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.139+0000 7f59ca7fc640 1 -- 172.21.15.98:0/1466217365 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 236997007 0 0) 0x7f59cc13ff10 con 0x7f59cc142a20 2024-03-25T18:16:17.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.139+0000 7f59ca7fc640 1 -- 172.21.15.98:0/1466217365 >> v1:172.21.15.97:6790/0 conn(0x7f59cc0a96b0 legacy=0x7f59cc13f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:17.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.139+0000 7f59ca7fc640 1 -- 172.21.15.98:0/1466217365 >> v1:172.21.15.98:6789/0 conn(0x7f59cc0b4700 legacy=0x7f59cc142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:17.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.139+0000 7f59ca7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:16:17.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.139+0000 7f59ca7fc640 1 monclient: found mon.noname-a 2024-03-25T18:16:17.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.139+0000 7f59ca7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:17.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.139+0000 7f59ca7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.97:6789/0 2024-03-25T18:16:17.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.139+0000 7f59ca7fc640 1 -- 172.21.15.98:0/1466217365 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59cc0b4ae0 con 0x7f59cc142a20 2024-03-25T18:16:17.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.139+0000 7f59ca7fc640 10 monclient: _finish_auth 0 2024-03-25T18:16:17.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.139+0000 7f59ca7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:16:17.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.139+0000 7f59ca7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:17.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.140+0000 7f59ca7fc640 1 -- 172.21.15.98:0/1466217365 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f59c00015b0 con 0x7f59cc142a20 2024-03-25T18:16:17.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.140+0000 7f59d1ca8640 5 monclient: authenticate success, global_id 4116 2024-03-25T18:16:17.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.140+0000 7f59d1ca8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:17.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.140+0000 7f59ca7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:17.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.140+0000 7f59ca7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:16:17.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.140+0000 7f59ca7fc640 10 monclient: dump: 2024-03-25T18:16:17.142 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-03-25T18:16:17.142 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:16:17.142 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:17.142 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:17.142 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:17.142 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-03-25T18:16:17.142 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:16:17.142 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:16:17.142 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:16:17.142 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-25T18:16:17.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.140+0000 7f59d1ca8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:17.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.141+0000 7f59ca7fc640 1 -- 172.21.15.98:0/1466217365 <== mon.0 v1:172.21.15.97:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f59c0001710 con 0x7f59cc142a20 2024-03-25T18:16:17.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.141+0000 7f59ca7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:17.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.141+0000 7f59ca7fc640 1 -- 172.21.15.98:0/1466217365 <== mon.0 v1:172.21.15.97:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f59c0001b60 con 0x7f59cc142a20 2024-03-25T18:16:17.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.141+0000 7f59ca7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:17.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.141+0000 7f59ca7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:16:17.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.141+0000 7f59ca7fc640 10 monclient: dump: 2024-03-25T18:16:17.144 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-03-25T18:16:17.144 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:16:17.144 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:17.144 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:17.144 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:17.144 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-03-25T18:16:17.145 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:16:17.145 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:16:17.145 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:16:17.145 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-25T18:16:17.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.141+0000 7f59d1ca8640 10 monclient: get_monmap_and_config success 2024-03-25T18:16:17.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.141+0000 7f59d1ca8640 4 set_mon_vals no callback set 2024-03-25T18:16:17.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.142+0000 7f59d1ca8640 10 monclient: shutdown 2024-03-25T18:16:17.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.142+0000 7f59d1ca8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:17.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.142+0000 7f59d1ca8640 1 -- 172.21.15.98:0/1466217365 >> v1:172.21.15.97:6789/0 conn(0x7f59cc142a20 legacy=0x7f59cc13f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:17.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.142+0000 7f59d1ca8640 1 -- 172.21.15.98:0/1466217365 shutdown_connections 2024-03-25T18:16:17.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.142+0000 7f59d1ca8640 1 -- 172.21.15.98:0/1466217365 >> 172.21.15.98:0/1466217365 conn(0x7f59cc0668e0 msgr2=0x7f59cc066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:17.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.142+0000 7f59d1ca8640 1 -- 172.21.15.98:0/1466217365 shutdown_connections 2024-03-25T18:16:17.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.142+0000 7f59d1ca8640 1 -- 172.21.15.98:0/1466217365 wait complete. 2024-03-25T18:16:17.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.143+0000 7f59d1ca8640 10 monclient: build_initial_monmap 2024-03-25T18:16:17.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.143+0000 7f59d1ca8640 10 monclient: monmap: 2024-03-25T18:16:17.146 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-03-25T18:16:17.146 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:16:17.146 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-03-25T18:16:17.146 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-03-25T18:16:17.146 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:17.146 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-03-25T18:16:17.146 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:16:17.146 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T18:16:17.146 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.97:6790/0 mon.noname-c 2024-03-25T18:16:17.146 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-25T18:16:17.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.143+0000 7f59d1ca8640 1 librados: starting msgr at 2024-03-25T18:16:17.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.143+0000 7f59d1ca8640 1 librados: starting objecter 2024-03-25T18:16:17.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.143+0000 7f59d1ca8640 1 Processor -- start 2024-03-25T18:16:17.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.143+0000 7f59d1ca8640 1 -- start start 2024-03-25T18:16:17.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.143+0000 7f59d1ca8640 1 librados: setting wanted keys 2024-03-25T18:16:17.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.143+0000 7f59d1ca8640 1 librados: calling monclient init 2024-03-25T18:16:17.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.143+0000 7f59d1ca8640 10 monclient: init 2024-03-25T18:16:17.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.144+0000 7f59d1ca8640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:17.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.144+0000 7f59d1ca8640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:16:17.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.144+0000 7f59d1ca8640 10 monclient(hunting): picked mon.noname-a con 0x7f59cc0a96b0 addr v1:172.21.15.97:6789/0 2024-03-25T18:16:17.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.144+0000 7f59d1ca8640 10 monclient(hunting): picked mon.noname-b con 0x7f59cc0b4700 addr v1:172.21.15.98:6789/0 2024-03-25T18:16:17.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.144+0000 7f59d1ca8640 10 monclient(hunting): picked mon.noname-c con 0x7f59cc142a20 addr v1:172.21.15.97:6790/0 2024-03-25T18:16:17.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.144+0000 7f59d1ca8640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59cc1317f0 con 0x7f59cc0a96b0 2024-03-25T18:16:17.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.144+0000 7f59d1ca8640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59cc13aaa0 con 0x7f59cc0b4700 2024-03-25T18:16:17.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.144+0000 7f59d1ca8640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59cc13ad80 con 0x7f59cc142a20 2024-03-25T18:16:17.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.144+0000 7f59d1ca8640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:17.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.144+0000 7f59d1ca8640 10 monclient(hunting): authenticate will time out at 7577.282197s 2024-03-25T18:16:17.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.144+0000 7f59cb7fe640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f59cc0a96b0 0x7f59cc13b480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.98:41042/0 (socket says 172.21.15.98:41042) 2024-03-25T18:16:17.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.144+0000 7f59cb7fe640 1 -- 172.21.15.98:0/3302956231 learned_addr learned my addr 172.21.15.98:0/3302956231 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-25T18:16:17.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.145+0000 7f59b3fff640 10 client.?.objecter ms_handle_connect 0x7f59cc0a96b0 2024-03-25T18:16:17.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.145+0000 7f59b3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:17.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.147+0000 7f59b3fff640 1 -- 172.21.15.98:0/3302956231 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1562103835 0 0) 0x7f59cc1317f0 con 0x7f59cc0a96b0 2024-03-25T18:16:17.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.147+0000 7f59b3fff640 10 monclient(hunting): _negotiate 2024-03-25T18:16:17.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.147+0000 7f59b3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:17.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.147+0000 7f59b3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:17.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.147+0000 7f59b3fff640 10 monclient(hunting): my global_id is 4119 2024-03-25T18:16:17.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.147+0000 7f59b3fff640 1 -- 172.21.15.98:0/3302956231 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59b8004eb0 con 0x7f59cc0a96b0 2024-03-25T18:16:17.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.154+0000 7f59b3fff640 1 -- 172.21.15.98:0/3302956231 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 931694778 0 0) 0x7f59b8004eb0 con 0x7f59cc0a96b0 2024-03-25T18:16:17.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.154+0000 7f59b3fff640 1 -- 172.21.15.98:0/3302956231 --> v1:172.21.15.97:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f59cc1317f0 con 0x7f59cc0a96b0 2024-03-25T18:16:17.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.154+0000 7f59b3fff640 1 -- 172.21.15.98:0/3302956231 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f59bc001970 con 0x7f59cc0a96b0 2024-03-25T18:16:17.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.154+0000 7f59b3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:17.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.154+0000 7f59b3fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:16:17.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.154+0000 7f59b3fff640 10 monclient(hunting): dump: 2024-03-25T18:16:17.155 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-03-25T18:16:17.155 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:16:17.155 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:17.155 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:17.155 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:17.155 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-03-25T18:16:17.156 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:16:17.156 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:16:17.156 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:16:17.156 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-25T18:16:17.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.154+0000 7f59b3fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:16:17.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.157+0000 7f59b3fff640 1 -- 172.21.15.98:0/3302956231 <== mon.0 v1:172.21.15.97:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 971863413 0 0) 0x7f59cc1317f0 con 0x7f59cc0a96b0 2024-03-25T18:16:17.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.157+0000 7f59b3fff640 1 -- 172.21.15.98:0/3302956231 >> v1:172.21.15.97:6790/0 conn(0x7f59cc142a20 legacy=0x7f59cc1310b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:17.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.157+0000 7f59b3fff640 1 -- 172.21.15.98:0/3302956231 >> v1:172.21.15.98:6789/0 conn(0x7f59cc0b4700 legacy=0x7f59cc1309a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:17.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.157+0000 7f59b3fff640 10 monclient: _finish_hunting 0 2024-03-25T18:16:17.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.157+0000 7f59b3fff640 1 monclient: found mon.a 2024-03-25T18:16:17.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.157+0000 7f59b3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:17.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.157+0000 7f59b3fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:16:17.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.157+0000 7f59b3fff640 1 -- 172.21.15.98:0/3302956231 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59cc13b060 con 0x7f59cc0a96b0 2024-03-25T18:16:17.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.157+0000 7f59b3fff640 10 monclient: _finish_auth 0 2024-03-25T18:16:17.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.157+0000 7f59b3fff640 10 monclient: _check_auth_tickets 2024-03-25T18:16:17.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.157+0000 7f59b3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:17.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.157+0000 7f59d1ca8640 5 monclient: authenticate success, global_id 4119 2024-03-25T18:16:17.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.157+0000 7f59d1ca8640 10 monclient: _renew_subs 2024-03-25T18:16:17.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.157+0000 7f59d1ca8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:16:17.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.157+0000 7f59d1ca8640 1 -- 172.21.15.98:0/3302956231 --> v1:172.21.15.97:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59cc139600 con 0x7f59cc0a96b0 2024-03-25T18:16:17.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.157+0000 7f59d1ca8640 10 client.4119.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:16:17.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.157+0000 7f59d1ca8640 10 monclient: _renew_subs 2024-03-25T18:16:17.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.157+0000 7f59d1ca8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:16:17.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.157+0000 7f59d1ca8640 1 -- 172.21.15.98:0/3302956231 --> v1:172.21.15.97:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f59cc1392a0 con 0x7f59cc0a96b0 2024-03-25T18:16:17.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.158+0000 7f59d1ca8640 1 librados: init done 2024-03-25T18:16:17.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.158+0000 7f59d1ca8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:17.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.158+0000 7f59d1ca8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:17.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.158+0000 7f59d1ca8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:16:17.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.158+0000 7f59d1ca8640 1 -- 172.21.15.98:0/3302956231 --> v1:172.21.15.97:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59cc04f730 con 0x7f59cc0a96b0 2024-03-25T18:16:17.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.162+0000 7f59b3fff640 1 -- 172.21.15.98:0/3302956231 <== mon.0 v1:172.21.15.97:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f59bc0047e0 con 0x7f59cc0a96b0 2024-03-25T18:16:17.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.162+0000 7f59b3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:17.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.162+0000 7f59b3fff640 1 -- 172.21.15.98:0/3302956231 <== mon.0 v1:172.21.15.97:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f59bc004b30 con 0x7f59cc0a96b0 2024-03-25T18:16:17.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.162+0000 7f59b3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:17.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.162+0000 7f59b3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:16:17.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.162+0000 7f59b3fff640 10 monclient: dump: 2024-03-25T18:16:17.163 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-03-25T18:16:17.163 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:16:17.163 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:17.163 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:17.163 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:17.163 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-03-25T18:16:17.163 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:16:17.163 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:16:17.163 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:16:17.163 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-25T18:16:17.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.162+0000 7f59c9ffb640 4 set_mon_vals no callback set 2024-03-25T18:16:17.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.162+0000 7f59b3fff640 1 -- 172.21.15.98:0/3302956231 <== mon.0 v1:172.21.15.97:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f59bc0050e0 con 0x7f59cc0a96b0 2024-03-25T18:16:17.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.162+0000 7f59b3fff640 1 -- 172.21.15.98:0/3302956231 <== mon.0 v1:172.21.15.97:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (unknown 424139732 0 0) 0x7f59bc0066a0 con 0x7f59cc0a96b0 2024-03-25T18:16:17.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.162+0000 7f59b3fff640 10 client.4119.objecter ms_dispatch 0x7f59cc000b90 osd_map(2..2 src has 1..2) v4 2024-03-25T18:16:17.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.162+0000 7f59b3fff640 3 client.4119.objecter handle_osd_map got epochs [2,2] > 0 2024-03-25T18:16:17.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.162+0000 7f59b3fff640 3 client.4119.objecter handle_osd_map decoding full epoch 2 2024-03-25T18:16:17.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.162+0000 7f59b3fff640 20 client.4119.objecter dump_active .. 0 homeless 2024-03-25T18:16:17.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.256+0000 7f59b3fff640 1 -- 172.21.15.98:0/3302956231 <== mon.0 v1:172.21.15.97:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f59bc005390 con 0x7f59cc0a96b0 2024-03-25T18:16:17.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.256+0000 7f59b3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:17.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.256+0000 7f59b3fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:16:17.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.328+0000 7f59d1ca8640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "cb5152b9-262f-45c3-94b8-18d0e81df558", "id": 0}] 2024-03-25T18:16:17.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.328+0000 7f59d1ca8640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "cb5152b9-262f-45c3-94b8-18d0e81df558", "id": 0}] 2024-03-25T18:16:17.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.328+0000 7f59d1ca8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:16:17.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.328+0000 7f59d1ca8640 1 -- 172.21.15.98:0/3302956231 --> v1:172.21.15.97:6789/0 -- mon_command({"prefix": "osd new", "uuid": "cb5152b9-262f-45c3-94b8-18d0e81df558", "id": 0} v 0) v1 -- 0x7f59cc0b7330 con 0x7f59cc0a96b0 2024-03-25T18:16:17.371 INFO:tasks.ceph.mgr.x.smithi098.stderr:2024-03-25T18:16:17.370+0000 7f16671db040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-25T18:16:17.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.418+0000 7f59b3fff640 1 -- 172.21.15.98:0/3302956231 <== mon.0 v1:172.21.15.97:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "cb5152b9-262f-45c3-94b8-18d0e81df558", "id": 0}]=0 v3) v1 ==== 112+0+2 (unknown 948164923 0 732707515) 0x7f59bc005680 con 0x7f59cc0a96b0 2024-03-25T18:16:17.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.419+0000 7f59b3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "cb5152b9-262f-45c3-94b8-18d0e81df558", "id": 0}] 2024-03-25T18:16:17.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.419+0000 7f59b3fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:16:17.420 INFO:teuthology.orchestra.run.smithi098.stdout:0 2024-03-25T18:16:17.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.419+0000 7f59d1ca8640 10 librados: watch_flush enter 2024-03-25T18:16:17.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.419+0000 7f59d1ca8640 10 librados: watch_flush exit 2024-03-25T18:16:17.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.419+0000 7f59d1ca8640 20 client.4119.objecter shutdown clearing up homeless session... 2024-03-25T18:16:17.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.419+0000 7f59d1ca8640 10 client.4119.objecter successfully canceled tick 2024-03-25T18:16:17.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.419+0000 7f59d1ca8640 10 monclient: shutdown 2024-03-25T18:16:17.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.419+0000 7f59d1ca8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:17.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.419+0000 7f59d1ca8640 1 -- 172.21.15.98:0/3302956231 >> v1:172.21.15.97:6789/0 conn(0x7f59cc0a96b0 legacy=0x7f59cc13b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:17.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.420+0000 7f59d1ca8640 1 -- 172.21.15.98:0/3302956231 shutdown_connections 2024-03-25T18:16:17.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.420+0000 7f59d1ca8640 1 -- 172.21.15.98:0/3302956231 >> 172.21.15.98:0/3302956231 conn(0x7f59cc0668e0 msgr2=0x7f59cc0b8590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:17.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.420+0000 7f59d1ca8640 1 -- 172.21.15.98:0/3302956231 shutdown_connections 2024-03-25T18:16:17.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.420+0000 7f59d1ca8640 1 -- 172.21.15.98:0/3302956231 wait complete. 2024-03-25T18:16:17.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.420+0000 7f59d1ca8640 1 librados: shutdown 2024-03-25T18:16:17.425 INFO:tasks.ceph.mgr.y.smithi097.stderr:2024-03-25T18:16:17.424+0000 7f7cdf810040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-25T18:16:17.427 INFO:tasks.ceph.mgr.x.smithi098.stderr:2024-03-25T18:16:17.426+0000 7f16671db040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-25T18:16:17.431 DEBUG:teuthology.orchestra.run.smithi098:> sudo ceph --cluster ceph osd new 6daf8793-0838-473d-a7d6-9a0d2894c75d 1 2024-03-25T18:16:17.476 INFO:tasks.ceph.mgr.x.smithi098.stderr:2024-03-25T18:16:17.475+0000 7f16671db040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-25T18:16:17.481 INFO:tasks.ceph.mgr.y.smithi097.stderr:2024-03-25T18:16:17.480+0000 7f7cdf810040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-25T18:16:17.526 INFO:tasks.ceph.mgr.y.smithi097.stderr:2024-03-25T18:16:17.524+0000 7f7cdf810040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-25T18:16:17.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.529+0000 7f305816d640 10 monclient: get_monmap_and_config 2024-03-25T18:16:17.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.529+0000 7f305816d640 10 monclient: build_initial_monmap 2024-03-25T18:16:17.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.530+0000 7f305816d640 10 monclient: monmap: 2024-03-25T18:16:17.531 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-03-25T18:16:17.531 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:16:17.531 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-03-25T18:16:17.530994+0000 2024-03-25T18:16:17.531 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-03-25T18:16:17.530994+0000 2024-03-25T18:16:17.531 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:17.531 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-03-25T18:16:17.531 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:16:17.531 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T18:16:17.531 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.97:6790/0 mon.noname-b 2024-03-25T18:16:17.532 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-25T18:16:17.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.530+0000 7f305816d640 1 Processor -- start 2024-03-25T18:16:17.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.530+0000 7f305816d640 1 -- start start 2024-03-25T18:16:17.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.530+0000 7f305816d640 10 monclient: init 2024-03-25T18:16:17.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.531+0000 7f305816d640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:17.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.531+0000 7f305816d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:16:17.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.531+0000 7f305816d640 10 monclient(hunting): picked mon.noname-a con 0x7f30500b4700 addr v1:172.21.15.97:6789/0 2024-03-25T18:16:17.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.531+0000 7f305816d640 10 monclient(hunting): picked mon.noname-c con 0x7f3050142a20 addr v1:172.21.15.98:6789/0 2024-03-25T18:16:17.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.531+0000 7f305816d640 10 monclient(hunting): picked mon.noname-b con 0x7f30500a96b0 addr v1:172.21.15.97:6790/0 2024-03-25T18:16:17.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.531+0000 7f305816d640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30500a9d60 con 0x7f30500b4700 2024-03-25T18:16:17.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.531+0000 7f305816d640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f305013ff10 con 0x7f3050142a20 2024-03-25T18:16:17.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.531+0000 7f305816d640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30500aa1f0 con 0x7f30500a96b0 2024-03-25T18:16:17.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.531+0000 7f305816d640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:17.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.531+0000 7f305816d640 10 monclient(hunting): authenticate will time out at 7577.669356s 2024-03-25T18:16:17.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.531+0000 7f305670a640 1 --1- >> v1:172.21.15.97:6790/0 conn(0x7f30500a96b0 0x7f305013f800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6790/0 says I am v1:172.21.15.98:45528/0 (socket says 172.21.15.98:45528) 2024-03-25T18:16:17.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.531+0000 7f305670a640 1 -- 172.21.15.98:0/175986071 learned_addr learned my addr 172.21.15.98:0/175986071 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-25T18:16:17.534 INFO:tasks.ceph.mgr.x.smithi098.stderr:2024-03-25T18:16:17.532+0000 7f16671db040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-25T18:16:17.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.534+0000 7f3054f07640 1 -- 172.21.15.98:0/175986071 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2695395102 0 0) 0x7f305013ff10 con 0x7f3050142a20 2024-03-25T18:16:17.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.534+0000 7f3054f07640 10 monclient(hunting): _negotiate 2024-03-25T18:16:17.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.534+0000 7f3054f07640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:17.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.534+0000 7f3054f07640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:17.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.534+0000 7f3054f07640 10 monclient(hunting): my global_id is 4111 2024-03-25T18:16:17.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.534+0000 7f3054f07640 1 -- 172.21.15.98:0/175986071 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f303c003520 con 0x7f3050142a20 2024-03-25T18:16:17.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.535+0000 7f3054f07640 1 -- 172.21.15.98:0/175986071 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3790067613 0 0) 0x7f30500a9d60 con 0x7f30500b4700 2024-03-25T18:16:17.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.535+0000 7f3054f07640 10 monclient(hunting): _negotiate 2024-03-25T18:16:17.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.535+0000 7f3054f07640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:17.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.535+0000 7f3054f07640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:17.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.535+0000 7f3054f07640 10 monclient(hunting): my global_id is 4122 2024-03-25T18:16:17.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.535+0000 7f3054f07640 1 -- 172.21.15.98:0/175986071 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f305013ff10 con 0x7f30500b4700 2024-03-25T18:16:17.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.536+0000 7f3054f07640 1 -- 172.21.15.98:0/175986071 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3945502950 0 0) 0x7f30500aa1f0 con 0x7f30500a96b0 2024-03-25T18:16:17.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.536+0000 7f3054f07640 10 monclient(hunting): _negotiate 2024-03-25T18:16:17.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.536+0000 7f3054f07640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:17.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.536+0000 7f3054f07640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:17.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.536+0000 7f3054f07640 10 monclient(hunting): my global_id is 4112 2024-03-25T18:16:17.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.536+0000 7f3054f07640 1 -- 172.21.15.98:0/175986071 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30500a9d60 con 0x7f30500a96b0 2024-03-25T18:16:17.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.539+0000 7f3054f07640 1 -- 172.21.15.98:0/175986071 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2191480203 0 0) 0x7f303c003520 con 0x7f3050142a20 2024-03-25T18:16:17.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.539+0000 7f3054f07640 1 -- 172.21.15.98:0/175986071 >> v1:172.21.15.97:6790/0 conn(0x7f30500a96b0 legacy=0x7f305013f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:17.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.539+0000 7f3054f07640 1 -- 172.21.15.98:0/175986071 >> v1:172.21.15.97:6789/0 conn(0x7f30500b4700 legacy=0x7f3050142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:17.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.539+0000 7f3054f07640 10 monclient: _finish_hunting 0 2024-03-25T18:16:17.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.539+0000 7f3054f07640 1 monclient: found mon.noname-c 2024-03-25T18:16:17.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.539+0000 7f3054f07640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:17.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.539+0000 7f3054f07640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.98:6789/0 2024-03-25T18:16:17.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.539+0000 7f3054f07640 1 -- 172.21.15.98:0/175986071 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30500b4ae0 con 0x7f3050142a20 2024-03-25T18:16:17.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.539+0000 7f3054f07640 10 monclient: _finish_auth 0 2024-03-25T18:16:17.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.539+0000 7f3054f07640 10 monclient: _check_auth_tickets 2024-03-25T18:16:17.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.539+0000 7f3054f07640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:17.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.539+0000 7f3054f07640 1 -- 172.21.15.98:0/175986071 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f304c001690 con 0x7f3050142a20 2024-03-25T18:16:17.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.539+0000 7f305816d640 5 monclient: authenticate success, global_id 4111 2024-03-25T18:16:17.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.539+0000 7f305816d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:17.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.539+0000 7f3054f07640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:17.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.539+0000 7f3054f07640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:16:17.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.539+0000 7f3054f07640 10 monclient: dump: 2024-03-25T18:16:17.541 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-03-25T18:16:17.541 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:16:17.541 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:17.541 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:17.541 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:17.541 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-03-25T18:16:17.541 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:16:17.541 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:16:17.541 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:16:17.541 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-25T18:16:17.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.539+0000 7f305816d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:17.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.567+0000 7f3054f07640 1 -- 172.21.15.98:0/175986071 <== mon.1 v1:172.21.15.98:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f304c0017f0 con 0x7f3050142a20 2024-03-25T18:16:17.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.567+0000 7f3054f07640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:17.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.567+0000 7f3054f07640 1 -- 172.21.15.98:0/175986071 <== mon.1 v1:172.21.15.98:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f304c001c00 con 0x7f3050142a20 2024-03-25T18:16:17.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.567+0000 7f3054f07640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:17.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.567+0000 7f3054f07640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:16:17.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.567+0000 7f3054f07640 10 monclient: dump: 2024-03-25T18:16:17.568 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-03-25T18:16:17.568 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:16:17.568 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:17.568 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:17.568 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:17.568 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-03-25T18:16:17.568 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:16:17.568 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:16:17.568 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:16:17.568 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-25T18:16:17.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.567+0000 7f305816d640 10 monclient: get_monmap_and_config success 2024-03-25T18:16:17.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.567+0000 7f305816d640 4 set_mon_vals no callback set 2024-03-25T18:16:17.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.567+0000 7f305816d640 10 monclient: shutdown 2024-03-25T18:16:17.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.567+0000 7f305816d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:17.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.567+0000 7f305816d640 1 -- 172.21.15.98:0/175986071 >> v1:172.21.15.98:6789/0 conn(0x7f3050142a20 legacy=0x7f305013f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:17.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.567+0000 7f305816d640 1 -- 172.21.15.98:0/175986071 shutdown_connections 2024-03-25T18:16:17.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.568+0000 7f305816d640 1 -- 172.21.15.98:0/175986071 >> 172.21.15.98:0/175986071 conn(0x7f30500668e0 msgr2=0x7f3050066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:17.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.568+0000 7f305816d640 1 -- 172.21.15.98:0/175986071 shutdown_connections 2024-03-25T18:16:17.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.568+0000 7f305816d640 1 -- 172.21.15.98:0/175986071 wait complete. 2024-03-25T18:16:17.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.568+0000 7f305816d640 10 monclient: build_initial_monmap 2024-03-25T18:16:17.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.568+0000 7f305816d640 10 monclient: monmap: 2024-03-25T18:16:17.571 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-03-25T18:16:17.572 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:16:17.572 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-03-25T18:16:17.572 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-03-25T18:16:17.572 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:17.572 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-03-25T18:16:17.572 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:16:17.572 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T18:16:17.572 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.97:6790/0 mon.noname-c 2024-03-25T18:16:17.572 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-25T18:16:17.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.569+0000 7f305816d640 1 librados: starting msgr at 2024-03-25T18:16:17.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.569+0000 7f305816d640 1 librados: starting objecter 2024-03-25T18:16:17.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.569+0000 7f305816d640 1 Processor -- start 2024-03-25T18:16:17.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.569+0000 7f305816d640 1 -- start start 2024-03-25T18:16:17.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.569+0000 7f305816d640 1 librados: setting wanted keys 2024-03-25T18:16:17.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.569+0000 7f305816d640 1 librados: calling monclient init 2024-03-25T18:16:17.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.569+0000 7f305816d640 10 monclient: init 2024-03-25T18:16:17.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.569+0000 7f305816d640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:17.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.569+0000 7f305816d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:16:17.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.569+0000 7f305816d640 10 monclient(hunting): picked mon.noname-c con 0x7f30500a96b0 addr v1:172.21.15.97:6790/0 2024-03-25T18:16:17.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.569+0000 7f305816d640 10 monclient(hunting): picked mon.noname-b con 0x7f30500b4700 addr v1:172.21.15.98:6789/0 2024-03-25T18:16:17.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.569+0000 7f305816d640 10 monclient(hunting): picked mon.noname-a con 0x7f3050142a20 addr v1:172.21.15.97:6789/0 2024-03-25T18:16:17.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.569+0000 7f305816d640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30500ae010 con 0x7f3050142a20 2024-03-25T18:16:17.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.569+0000 7f305816d640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30500ae2f0 con 0x7f30500b4700 2024-03-25T18:16:17.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.569+0000 7f305816d640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30500af100 con 0x7f30500a96b0 2024-03-25T18:16:17.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.569+0000 7f305816d640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:17.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.569+0000 7f305816d640 10 monclient(hunting): authenticate will time out at 7577.707711s 2024-03-25T18:16:17.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.570+0000 7f3055f09640 1 --1- >> v1:172.21.15.97:6790/0 conn(0x7f30500a96b0 0x7f30500ae9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.97:6790/0 says I am v1:172.21.15.98:45542/0 (socket says 172.21.15.98:45542) 2024-03-25T18:16:17.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.570+0000 7f3055f09640 1 -- 172.21.15.98:0/2890858874 learned_addr learned my addr 172.21.15.98:0/2890858874 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-25T18:16:17.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.570+0000 7f30427fc640 10 client.?.objecter ms_handle_connect 0x7f30500a96b0 2024-03-25T18:16:17.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.570+0000 7f30427fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:17.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.570+0000 7f30427fc640 10 client.?.objecter ms_handle_connect 0x7f3050142a20 2024-03-25T18:16:17.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.570+0000 7f30427fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:17.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.571+0000 7f30427fc640 10 client.?.objecter ms_handle_connect 0x7f30500b4700 2024-03-25T18:16:17.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.571+0000 7f30427fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:17.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.572+0000 7f30427fc640 1 -- 172.21.15.98:0/2890858874 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1589966109 0 0) 0x7f30500af100 con 0x7f30500a96b0 2024-03-25T18:16:17.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.572+0000 7f30427fc640 10 monclient(hunting): _negotiate 2024-03-25T18:16:17.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.572+0000 7f30427fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:17.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.572+0000 7f30427fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:17.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.572+0000 7f30427fc640 10 monclient(hunting): my global_id is 4115 2024-03-25T18:16:17.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.572+0000 7f30427fc640 1 -- 172.21.15.98:0/2890858874 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3034004eb0 con 0x7f30500a96b0 2024-03-25T18:16:17.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.573+0000 7f30427fc640 1 -- 172.21.15.98:0/2890858874 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2713977469 0 0) 0x7f30500ae010 con 0x7f3050142a20 2024-03-25T18:16:17.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.573+0000 7f30427fc640 10 monclient(hunting): _negotiate 2024-03-25T18:16:17.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.573+0000 7f30427fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:17.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.573+0000 7f30427fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:17.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.573+0000 7f30427fc640 10 monclient(hunting): my global_id is 4125 2024-03-25T18:16:17.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.573+0000 7f30427fc640 1 -- 172.21.15.98:0/2890858874 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30500af100 con 0x7f3050142a20 2024-03-25T18:16:17.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.573+0000 7f30427fc640 1 -- 172.21.15.98:0/2890858874 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2471720356 0 0) 0x7f30500ae2f0 con 0x7f30500b4700 2024-03-25T18:16:17.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.573+0000 7f30427fc640 10 monclient(hunting): _negotiate 2024-03-25T18:16:17.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.573+0000 7f30427fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:17.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.573+0000 7f30427fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:17.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.573+0000 7f30427fc640 10 monclient(hunting): my global_id is 4114 2024-03-25T18:16:17.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.573+0000 7f30427fc640 1 -- 172.21.15.98:0/2890858874 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30500ae010 con 0x7f30500b4700 2024-03-25T18:16:17.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.578+0000 7f30427fc640 1 -- 172.21.15.98:0/2890858874 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 172901362 0 0) 0x7f30500af100 con 0x7f3050142a20 2024-03-25T18:16:17.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.578+0000 7f30427fc640 1 -- 172.21.15.98:0/2890858874 --> v1:172.21.15.97:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f30500ae2f0 con 0x7f3050142a20 2024-03-25T18:16:17.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.578+0000 7f30427fc640 1 -- 172.21.15.98:0/2890858874 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f3048001b50 con 0x7f3050142a20 2024-03-25T18:16:17.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.578+0000 7f30427fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:17.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.578+0000 7f30427fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:16:17.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.578+0000 7f30427fc640 10 monclient(hunting): dump: 2024-03-25T18:16:17.579 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-03-25T18:16:17.579 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:16:17.579 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:17.579 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:17.579 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:17.579 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-03-25T18:16:17.579 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:16:17.579 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:16:17.579 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:16:17.579 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-25T18:16:17.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.578+0000 7f30427fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:16:17.580 INFO:tasks.ceph.mgr.x.smithi098.stderr:2024-03-25T18:16:17.578+0000 7f16671db040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-25T18:16:17.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.581+0000 7f30427fc640 1 -- 172.21.15.98:0/2890858874 <== mon.0 v1:172.21.15.97:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4284816314 0 0) 0x7f30500ae2f0 con 0x7f3050142a20 2024-03-25T18:16:17.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.581+0000 7f30427fc640 1 -- 172.21.15.98:0/2890858874 >> v1:172.21.15.97:6790/0 conn(0x7f30500a96b0 legacy=0x7f30500ae9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:17.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.581+0000 7f30427fc640 1 -- 172.21.15.98:0/2890858874 >> v1:172.21.15.98:6789/0 conn(0x7f30500b4700 legacy=0x7f3050139e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:17.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.581+0000 7f30427fc640 10 monclient: _finish_hunting 0 2024-03-25T18:16:17.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.581+0000 7f30427fc640 1 monclient: found mon.a 2024-03-25T18:16:17.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.581+0000 7f30427fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:17.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.581+0000 7f30427fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:16:17.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.581+0000 7f30427fc640 1 -- 172.21.15.98:0/2890858874 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30500ad120 con 0x7f3050142a20 2024-03-25T18:16:17.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.581+0000 7f30427fc640 10 monclient: _finish_auth 0 2024-03-25T18:16:17.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.581+0000 7f30427fc640 10 monclient: _check_auth_tickets 2024-03-25T18:16:17.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.581+0000 7f30427fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:17.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.581+0000 7f305816d640 5 monclient: authenticate success, global_id 4125 2024-03-25T18:16:17.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.581+0000 7f305816d640 10 monclient: _renew_subs 2024-03-25T18:16:17.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.581+0000 7f305816d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:16:17.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.581+0000 7f305816d640 1 -- 172.21.15.98:0/2890858874 --> v1:172.21.15.97:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30500ad3f0 con 0x7f3050142a20 2024-03-25T18:16:17.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.581+0000 7f305816d640 10 client.4125.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:16:17.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.581+0000 7f305816d640 10 monclient: _renew_subs 2024-03-25T18:16:17.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.581+0000 7f305816d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:16:17.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.581+0000 7f305816d640 1 -- 172.21.15.98:0/2890858874 --> v1:172.21.15.97:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f30500ac9c0 con 0x7f3050142a20 2024-03-25T18:16:17.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.581+0000 7f305816d640 1 librados: init done 2024-03-25T18:16:17.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.582+0000 7f305816d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:17.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.582+0000 7f305816d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:17.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.582+0000 7f305816d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:16:17.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.582+0000 7f305816d640 1 -- 172.21.15.98:0/2890858874 --> v1:172.21.15.97:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f305004f730 con 0x7f3050142a20 2024-03-25T18:16:17.585 INFO:tasks.ceph.mgr.y.smithi097.stderr:2024-03-25T18:16:17.583+0000 7f7cdf810040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-25T18:16:17.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.586+0000 7f30427fc640 1 -- 172.21.15.98:0/2890858874 <== mon.0 v1:172.21.15.97:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3048001f60 con 0x7f3050142a20 2024-03-25T18:16:17.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.586+0000 7f30427fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:17.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.586+0000 7f30427fc640 1 -- 172.21.15.98:0/2890858874 <== mon.0 v1:172.21.15.97:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f3048004d00 con 0x7f3050142a20 2024-03-25T18:16:17.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.586+0000 7f30427fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:17.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.586+0000 7f30427fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:16:17.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.586+0000 7f30427fc640 10 monclient: dump: 2024-03-25T18:16:17.587 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-03-25T18:16:17.587 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:16:17.588 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:17.588 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:17.588 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:17.588 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-03-25T18:16:17.588 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:16:17.588 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:16:17.588 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:16:17.588 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-25T18:16:17.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.586+0000 7f3043fff640 4 set_mon_vals no callback set 2024-03-25T18:16:17.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.586+0000 7f30427fc640 1 -- 172.21.15.98:0/2890858874 <== mon.0 v1:172.21.15.97:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f30480052b0 con 0x7f3050142a20 2024-03-25T18:16:17.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.586+0000 7f30427fc640 1 -- 172.21.15.98:0/2890858874 <== mon.0 v1:172.21.15.97:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (unknown 3477074874 0 0) 0x7f3048006810 con 0x7f3050142a20 2024-03-25T18:16:17.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.586+0000 7f30427fc640 10 client.4125.objecter ms_dispatch 0x7f3050000b90 osd_map(3..3 src has 1..3) v4 2024-03-25T18:16:17.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.586+0000 7f30427fc640 3 client.4125.objecter handle_osd_map got epochs [3,3] > 0 2024-03-25T18:16:17.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.586+0000 7f30427fc640 3 client.4125.objecter handle_osd_map decoding full epoch 3 2024-03-25T18:16:17.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.586+0000 7f30427fc640 20 client.4125.objecter dump_active .. 0 homeless 2024-03-25T18:16:17.629 INFO:tasks.ceph.mgr.y.smithi097.stderr:2024-03-25T18:16:17.628+0000 7f7cdf810040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-25T18:16:17.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.685+0000 7f30427fc640 1 -- 172.21.15.98:0/2890858874 <== mon.0 v1:172.21.15.97:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f3048005c50 con 0x7f3050142a20 2024-03-25T18:16:17.685 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.685+0000 7f30427fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:17.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.685+0000 7f30427fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:16:17.695 INFO:tasks.ceph.mgr.x.smithi098.stderr:2024-03-25T18:16:17.695+0000 7f16671db040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-25T18:16:17.746 INFO:tasks.ceph.mgr.y.smithi097.stderr:2024-03-25T18:16:17.745+0000 7f7cdf810040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-25T18:16:17.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.758+0000 7f305816d640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "6daf8793-0838-473d-a7d6-9a0d2894c75d", "id": 1}] 2024-03-25T18:16:17.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.758+0000 7f305816d640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "6daf8793-0838-473d-a7d6-9a0d2894c75d", "id": 1}] 2024-03-25T18:16:17.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.758+0000 7f305816d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:16:17.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.758+0000 7f305816d640 1 -- 172.21.15.98:0/2890858874 --> v1:172.21.15.97:6789/0 -- mon_command({"prefix": "osd new", "uuid": "6daf8793-0838-473d-a7d6-9a0d2894c75d", "id": 1} v 0) v1 -- 0x7f3050142220 con 0x7f3050142a20 2024-03-25T18:16:17.767 INFO:tasks.ceph.mgr.x.smithi098.stderr:2024-03-25T18:16:17.766+0000 7f16671db040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-25T18:16:17.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.792+0000 7f30427fc640 1 -- 172.21.15.98:0/2890858874 <== mon.0 v1:172.21.15.97:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "6daf8793-0838-473d-a7d6-9a0d2894c75d", "id": 1}]=0 v4) v1 ==== 112+0+2 (unknown 2627770229 0 940483276) 0x7f3048005ed0 con 0x7f3050142a20 2024-03-25T18:16:17.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.792+0000 7f30427fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "6daf8793-0838-473d-a7d6-9a0d2894c75d", "id": 1}] 2024-03-25T18:16:17.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.792+0000 7f30427fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:16:17.793 INFO:teuthology.orchestra.run.smithi098.stdout:1 2024-03-25T18:16:17.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.793+0000 7f305816d640 10 librados: watch_flush enter 2024-03-25T18:16:17.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.793+0000 7f305816d640 10 librados: watch_flush exit 2024-03-25T18:16:17.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.793+0000 7f305816d640 20 client.4125.objecter shutdown clearing up homeless session... 2024-03-25T18:16:17.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.793+0000 7f305816d640 10 client.4125.objecter successfully canceled tick 2024-03-25T18:16:17.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.793+0000 7f305816d640 10 monclient: shutdown 2024-03-25T18:16:17.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.793+0000 7f305816d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:17.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.793+0000 7f305816d640 1 -- 172.21.15.98:0/2890858874 >> v1:172.21.15.97:6789/0 conn(0x7f3050142a20 legacy=0x7f305013a530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:17.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.794+0000 7f305816d640 1 -- 172.21.15.98:0/2890858874 shutdown_connections 2024-03-25T18:16:17.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.794+0000 7f305816d640 1 -- 172.21.15.98:0/2890858874 >> 172.21.15.98:0/2890858874 conn(0x7f30500668e0 msgr2=0x7f305013c2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:17.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.794+0000 7f305816d640 1 -- 172.21.15.98:0/2890858874 shutdown_connections 2024-03-25T18:16:17.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.794+0000 7f305816d640 1 -- 172.21.15.98:0/2890858874 wait complete. 2024-03-25T18:16:17.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.794+0000 7f305816d640 1 librados: shutdown 2024-03-25T18:16:17.807 DEBUG:teuthology.orchestra.run.smithi098:> sudo ceph --cluster ceph osd new 136f1493-3b9c-4f43-a2e7-641482d69dae 2 2024-03-25T18:16:17.816 INFO:tasks.ceph.mgr.x.smithi098.stderr:2024-03-25T18:16:17.815+0000 7f16671db040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-25T18:16:17.820 INFO:tasks.ceph.mgr.y.smithi097.stderr:2024-03-25T18:16:17.819+0000 7f7cdf810040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-25T18:16:17.872 INFO:tasks.ceph.mgr.x.smithi098.stderr:2024-03-25T18:16:17.871+0000 7f16671db040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-25T18:16:17.878 INFO:tasks.ceph.mgr.y.smithi097.stderr:2024-03-25T18:16:17.876+0000 7f7cdf810040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-25T18:16:17.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.908+0000 7f39efe93640 10 monclient: get_monmap_and_config 2024-03-25T18:16:17.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.908+0000 7f39efe93640 10 monclient: build_initial_monmap 2024-03-25T18:16:17.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.908+0000 7f39efe93640 10 monclient: monmap: 2024-03-25T18:16:17.910 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-03-25T18:16:17.910 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:16:17.910 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-03-25T18:16:17.909960+0000 2024-03-25T18:16:17.910 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-03-25T18:16:17.909960+0000 2024-03-25T18:16:17.910 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:17.910 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-03-25T18:16:17.910 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:16:17.910 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T18:16:17.910 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.97:6790/0 mon.noname-b 2024-03-25T18:16:17.910 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-25T18:16:17.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.909+0000 7f39efe93640 1 Processor -- start 2024-03-25T18:16:17.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.909+0000 7f39efe93640 1 -- start start 2024-03-25T18:16:17.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.909+0000 7f39efe93640 10 monclient: init 2024-03-25T18:16:17.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.910+0000 7f39efe93640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:17.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.910+0000 7f39efe93640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:16:17.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.910+0000 7f39efe93640 10 monclient(hunting): picked mon.noname-c con 0x7f39e80b4700 addr v1:172.21.15.98:6789/0 2024-03-25T18:16:17.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.910+0000 7f39efe93640 10 monclient(hunting): picked mon.noname-b con 0x7f39e8142a20 addr v1:172.21.15.97:6790/0 2024-03-25T18:16:17.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.910+0000 7f39efe93640 10 monclient(hunting): picked mon.noname-a con 0x7f39e80a96b0 addr v1:172.21.15.97:6789/0 2024-03-25T18:16:17.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.910+0000 7f39efe93640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39e80a9d60 con 0x7f39e80a96b0 2024-03-25T18:16:17.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.910+0000 7f39efe93640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39e813ff10 con 0x7f39e80b4700 2024-03-25T18:16:17.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.910+0000 7f39efe93640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39e80aa1f0 con 0x7f39e8142a20 2024-03-25T18:16:17.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.910+0000 7f39efe93640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:17.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.910+0000 7f39efe93640 10 monclient(hunting): authenticate will time out at 7578.048234s 2024-03-25T18:16:17.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.910+0000 7f39ed42e640 1 --1- >> v1:172.21.15.97:6790/0 conn(0x7f39e8142a20 0x7f39e813f0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6790/0 says I am v1:172.21.15.98:45548/0 (socket says 172.21.15.98:45548) 2024-03-25T18:16:17.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.910+0000 7f39ed42e640 1 -- 172.21.15.98:0/443773712 learned_addr learned my addr 172.21.15.98:0/443773712 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-25T18:16:17.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.913+0000 7f39ecc2d640 1 -- 172.21.15.98:0/443773712 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2685681228 0 0) 0x7f39e80aa1f0 con 0x7f39e8142a20 2024-03-25T18:16:17.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.913+0000 7f39ecc2d640 10 monclient(hunting): _negotiate 2024-03-25T18:16:17.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.913+0000 7f39ecc2d640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:17.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.913+0000 7f39ecc2d640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:17.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.913+0000 7f39ecc2d640 10 monclient(hunting): my global_id is 4118 2024-03-25T18:16:17.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.913+0000 7f39ecc2d640 1 -- 172.21.15.98:0/443773712 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39d4003520 con 0x7f39e8142a20 2024-03-25T18:16:17.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.913+0000 7f39ecc2d640 1 -- 172.21.15.98:0/443773712 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2123429048 0 0) 0x7f39e813ff10 con 0x7f39e80b4700 2024-03-25T18:16:17.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.913+0000 7f39ecc2d640 10 monclient(hunting): _negotiate 2024-03-25T18:16:17.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.913+0000 7f39ecc2d640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:17.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.913+0000 7f39ecc2d640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:17.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.913+0000 7f39ecc2d640 10 monclient(hunting): my global_id is 4117 2024-03-25T18:16:17.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.913+0000 7f39ecc2d640 1 -- 172.21.15.98:0/443773712 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39e80aa1f0 con 0x7f39e80b4700 2024-03-25T18:16:17.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.913+0000 7f39ecc2d640 1 -- 172.21.15.98:0/443773712 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 307708841 0 0) 0x7f39e80a9d60 con 0x7f39e80a96b0 2024-03-25T18:16:17.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.913+0000 7f39ecc2d640 10 monclient(hunting): _negotiate 2024-03-25T18:16:17.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.913+0000 7f39ecc2d640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:17.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.913+0000 7f39ecc2d640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:17.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.913+0000 7f39ecc2d640 10 monclient(hunting): my global_id is 4128 2024-03-25T18:16:17.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.913+0000 7f39ecc2d640 1 -- 172.21.15.98:0/443773712 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39e813ff10 con 0x7f39e80a96b0 2024-03-25T18:16:17.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.916+0000 7f39ecc2d640 1 -- 172.21.15.98:0/443773712 <== mon.2 v1:172.21.15.97:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1875146221 0 0) 0x7f39d4003520 con 0x7f39e8142a20 2024-03-25T18:16:17.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.916+0000 7f39ecc2d640 1 -- 172.21.15.98:0/443773712 >> v1:172.21.15.98:6789/0 conn(0x7f39e80b4700 legacy=0x7f39e8142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:17.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.916+0000 7f39ecc2d640 1 -- 172.21.15.98:0/443773712 >> v1:172.21.15.97:6789/0 conn(0x7f39e80a96b0 legacy=0x7f39e813f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:17.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.916+0000 7f39ecc2d640 10 monclient: _finish_hunting 0 2024-03-25T18:16:17.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.916+0000 7f39ecc2d640 1 monclient: found mon.noname-b 2024-03-25T18:16:17.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.916+0000 7f39ecc2d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:17.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.916+0000 7f39ecc2d640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.97:6790/0 2024-03-25T18:16:17.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.916+0000 7f39ecc2d640 1 -- 172.21.15.98:0/443773712 --> v1:172.21.15.97:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39e80b4ae0 con 0x7f39e8142a20 2024-03-25T18:16:17.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.916+0000 7f39ecc2d640 10 monclient: _finish_auth 0 2024-03-25T18:16:17.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.916+0000 7f39ecc2d640 10 monclient: _check_auth_tickets 2024-03-25T18:16:17.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.916+0000 7f39ecc2d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:17.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.916+0000 7f39ecc2d640 1 -- 172.21.15.98:0/443773712 <== mon.2 v1:172.21.15.97:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f39d8001690 con 0x7f39e8142a20 2024-03-25T18:16:17.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.916+0000 7f39efe93640 5 monclient: authenticate success, global_id 4118 2024-03-25T18:16:17.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.916+0000 7f39efe93640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:17.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.916+0000 7f39ecc2d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:17.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.916+0000 7f39ecc2d640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:16:17.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.916+0000 7f39ecc2d640 10 monclient: dump: 2024-03-25T18:16:17.919 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-03-25T18:16:17.919 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:16:17.919 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:17.919 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:17.919 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:17.919 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-03-25T18:16:17.919 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:16:17.919 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:16:17.919 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:16:17.919 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-25T18:16:17.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.916+0000 7f39efe93640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:17.922 INFO:tasks.ceph.mgr.x.smithi098.stderr:2024-03-25T18:16:17.921+0000 7f16671db040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-25T18:16:17.935 INFO:tasks.ceph.mgr.y.smithi097.stderr:2024-03-25T18:16:17.934+0000 7f7cdf810040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-25T18:16:17.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.942+0000 7f39ecc2d640 1 -- 172.21.15.98:0/443773712 <== mon.2 v1:172.21.15.97:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f39d80017f0 con 0x7f39e8142a20 2024-03-25T18:16:17.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.942+0000 7f39ecc2d640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:17.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.942+0000 7f39ecc2d640 1 -- 172.21.15.98:0/443773712 <== mon.2 v1:172.21.15.97:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f39d8001c00 con 0x7f39e8142a20 2024-03-25T18:16:17.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.942+0000 7f39ecc2d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:17.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.942+0000 7f39ecc2d640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:16:17.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.942+0000 7f39ecc2d640 10 monclient: dump: 2024-03-25T18:16:17.943 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-03-25T18:16:17.944 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:16:17.944 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:17.944 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:17.944 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:17.944 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-03-25T18:16:17.944 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:16:17.944 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:16:17.944 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:16:17.944 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-25T18:16:17.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.942+0000 7f39efe93640 10 monclient: get_monmap_and_config success 2024-03-25T18:16:17.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.942+0000 7f39efe93640 4 set_mon_vals no callback set 2024-03-25T18:16:17.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.943+0000 7f39efe93640 10 monclient: shutdown 2024-03-25T18:16:17.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.943+0000 7f39efe93640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:17.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.943+0000 7f39efe93640 1 -- 172.21.15.98:0/443773712 >> v1:172.21.15.97:6790/0 conn(0x7f39e8142a20 legacy=0x7f39e813f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:17.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.943+0000 7f39efe93640 1 -- 172.21.15.98:0/443773712 shutdown_connections 2024-03-25T18:16:17.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.943+0000 7f39efe93640 1 -- 172.21.15.98:0/443773712 >> 172.21.15.98:0/443773712 conn(0x7f39e80668e0 msgr2=0x7f39e8066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:17.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.943+0000 7f39efe93640 1 -- 172.21.15.98:0/443773712 shutdown_connections 2024-03-25T18:16:17.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.943+0000 7f39efe93640 1 -- 172.21.15.98:0/443773712 wait complete. 2024-03-25T18:16:17.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.944+0000 7f39efe93640 10 monclient: build_initial_monmap 2024-03-25T18:16:17.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.944+0000 7f39efe93640 10 monclient: monmap: 2024-03-25T18:16:17.947 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-03-25T18:16:17.947 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:16:17.947 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-03-25T18:16:17.947 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-03-25T18:16:17.947 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:17.948 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-03-25T18:16:17.948 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:16:17.948 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T18:16:17.948 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.97:6790/0 mon.noname-c 2024-03-25T18:16:17.948 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-25T18:16:17.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.944+0000 7f39efe93640 1 librados: starting msgr at 2024-03-25T18:16:17.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.944+0000 7f39efe93640 1 librados: starting objecter 2024-03-25T18:16:17.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.944+0000 7f39efe93640 1 Processor -- start 2024-03-25T18:16:17.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.944+0000 7f39efe93640 1 -- start start 2024-03-25T18:16:17.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.944+0000 7f39efe93640 1 librados: setting wanted keys 2024-03-25T18:16:17.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.944+0000 7f39efe93640 1 librados: calling monclient init 2024-03-25T18:16:17.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.944+0000 7f39efe93640 10 monclient: init 2024-03-25T18:16:17.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.945+0000 7f39efe93640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:17.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.945+0000 7f39efe93640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:16:17.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.945+0000 7f39efe93640 10 monclient(hunting): picked mon.noname-c con 0x7f39e80a96b0 addr v1:172.21.15.97:6790/0 2024-03-25T18:16:17.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.945+0000 7f39efe93640 10 monclient(hunting): picked mon.noname-a con 0x7f39e80b4700 addr v1:172.21.15.97:6789/0 2024-03-25T18:16:17.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.945+0000 7f39efe93640 10 monclient(hunting): picked mon.noname-b con 0x7f39e8142a20 addr v1:172.21.15.98:6789/0 2024-03-25T18:16:17.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.945+0000 7f39efe93640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39e80ae500 con 0x7f39e80b4700 2024-03-25T18:16:17.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.945+0000 7f39efe93640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39e80ae7e0 con 0x7f39e8142a20 2024-03-25T18:16:17.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.945+0000 7f39efe93640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39e81dbad0 con 0x7f39e80a96b0 2024-03-25T18:16:17.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.945+0000 7f39efe93640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:17.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.945+0000 7f39efe93640 10 monclient(hunting): authenticate will time out at 7578.083031s 2024-03-25T18:16:17.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.945+0000 7f39ee430640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f39e8142a20 0x7f39e81e23e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:60666/0 (socket says 172.21.15.98:60666) 2024-03-25T18:16:17.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.945+0000 7f39ee430640 1 -- 172.21.15.98:0/2322273063 learned_addr learned my addr 172.21.15.98:0/2322273063 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-25T18:16:17.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.946+0000 7f39de7fc640 10 client.?.objecter ms_handle_connect 0x7f39e8142a20 2024-03-25T18:16:17.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.946+0000 7f39de7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:17.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.946+0000 7f39de7fc640 10 client.?.objecter ms_handle_connect 0x7f39e80a96b0 2024-03-25T18:16:17.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.946+0000 7f39de7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:17.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.946+0000 7f39de7fc640 10 client.?.objecter ms_handle_connect 0x7f39e80b4700 2024-03-25T18:16:17.950 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.946+0000 7f39de7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:17.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.947+0000 7f39de7fc640 1 -- 172.21.15.98:0/2322273063 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2677536958 0 0) 0x7f39e80ae7e0 con 0x7f39e8142a20 2024-03-25T18:16:17.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.948+0000 7f39de7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:16:17.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.948+0000 7f39de7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:17.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.948+0000 7f39de7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:17.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.948+0000 7f39de7fc640 10 monclient(hunting): my global_id is 4120 2024-03-25T18:16:17.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.948+0000 7f39de7fc640 1 -- 172.21.15.98:0/2322273063 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39cc004eb0 con 0x7f39e8142a20 2024-03-25T18:16:17.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.948+0000 7f39de7fc640 1 -- 172.21.15.98:0/2322273063 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3869477027 0 0) 0x7f39e81dbad0 con 0x7f39e80a96b0 2024-03-25T18:16:17.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.948+0000 7f39de7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:16:17.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.948+0000 7f39de7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:17.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.948+0000 7f39de7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:17.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.948+0000 7f39de7fc640 10 monclient(hunting): my global_id is 4121 2024-03-25T18:16:17.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.948+0000 7f39de7fc640 1 -- 172.21.15.98:0/2322273063 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39e80ae7e0 con 0x7f39e80a96b0 2024-03-25T18:16:17.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.948+0000 7f39de7fc640 1 -- 172.21.15.98:0/2322273063 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3286192360 0 0) 0x7f39e80ae500 con 0x7f39e80b4700 2024-03-25T18:16:17.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.948+0000 7f39de7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:16:17.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.948+0000 7f39de7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:17.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.948+0000 7f39de7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:17.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.948+0000 7f39de7fc640 10 monclient(hunting): my global_id is 4131 2024-03-25T18:16:17.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.948+0000 7f39de7fc640 1 -- 172.21.15.98:0/2322273063 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39e81dbad0 con 0x7f39e80b4700 2024-03-25T18:16:17.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.952+0000 7f39de7fc640 1 -- 172.21.15.98:0/2322273063 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1787658258 0 0) 0x7f39cc004eb0 con 0x7f39e8142a20 2024-03-25T18:16:17.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.952+0000 7f39de7fc640 1 -- 172.21.15.98:0/2322273063 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f39e80ae500 con 0x7f39e8142a20 2024-03-25T18:16:17.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.952+0000 7f39de7fc640 1 -- 172.21.15.98:0/2322273063 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f39e4001d30 con 0x7f39e8142a20 2024-03-25T18:16:17.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.952+0000 7f39de7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:17.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.952+0000 7f39de7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:16:17.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.952+0000 7f39de7fc640 10 monclient(hunting): dump: 2024-03-25T18:16:17.954 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-03-25T18:16:17.954 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:16:17.954 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:17.954 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:17.954 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:17.955 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-03-25T18:16:17.955 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:16:17.955 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:16:17.955 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:16:17.955 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-25T18:16:17.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.952+0000 7f39de7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:16:17.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.952+0000 7f39de7fc640 1 -- 172.21.15.98:0/2322273063 <== mon.2 v1:172.21.15.97:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 191980426 0 0) 0x7f39e80ae7e0 con 0x7f39e80a96b0 2024-03-25T18:16:17.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.952+0000 7f39de7fc640 1 -- 172.21.15.98:0/2322273063 --> v1:172.21.15.97:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f39cc004eb0 con 0x7f39e80a96b0 2024-03-25T18:16:17.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.952+0000 7f39de7fc640 1 -- 172.21.15.98:0/2322273063 <== mon.2 v1:172.21.15.97:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f39e0001970 con 0x7f39e80a96b0 2024-03-25T18:16:17.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.952+0000 7f39de7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:17.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.952+0000 7f39de7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:16:17.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.952+0000 7f39de7fc640 10 monclient(hunting): dump: 2024-03-25T18:16:17.955 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-03-25T18:16:17.956 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:16:17.956 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:17.956 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:17.956 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:17.956 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-03-25T18:16:17.956 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:16:17.956 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:16:17.956 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:16:17.956 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-25T18:16:17.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.953+0000 7f39de7fc640 1 -- 172.21.15.98:0/2322273063 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2511171144 0 0) 0x7f39e81dbad0 con 0x7f39e80b4700 2024-03-25T18:16:17.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.953+0000 7f39de7fc640 1 -- 172.21.15.98:0/2322273063 --> v1:172.21.15.97:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f39e80ae7e0 con 0x7f39e80b4700 2024-03-25T18:16:17.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.953+0000 7f39de7fc640 1 -- 172.21.15.98:0/2322273063 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f39d8001b00 con 0x7f39e80b4700 2024-03-25T18:16:17.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.953+0000 7f39de7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:17.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.953+0000 7f39de7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:16:17.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.953+0000 7f39de7fc640 10 monclient(hunting): dump: 2024-03-25T18:16:17.956 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-03-25T18:16:17.957 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:16:17.957 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:17.957 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:17.957 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:17.957 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-03-25T18:16:17.957 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:16:17.957 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:16:17.957 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:16:17.957 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-25T18:16:17.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.956+0000 7f39de7fc640 1 -- 172.21.15.98:0/2322273063 <== mon.0 v1:172.21.15.97:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3752844624 0 0) 0x7f39e80ae7e0 con 0x7f39e80b4700 2024-03-25T18:16:17.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.956+0000 7f39de7fc640 1 -- 172.21.15.98:0/2322273063 >> v1:172.21.15.97:6790/0 conn(0x7f39e80a96b0 legacy=0x7f39e81db320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:17.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.956+0000 7f39de7fc640 1 -- 172.21.15.98:0/2322273063 >> v1:172.21.15.98:6789/0 conn(0x7f39e8142a20 legacy=0x7f39e81e23e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:17.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.956+0000 7f39de7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:16:17.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.956+0000 7f39de7fc640 1 monclient: found mon.a 2024-03-25T18:16:17.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.956+0000 7f39de7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:17.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.956+0000 7f39de7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:16:17.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.956+0000 7f39de7fc640 1 -- 172.21.15.98:0/2322273063 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39e813a560 con 0x7f39e80b4700 2024-03-25T18:16:17.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.956+0000 7f39de7fc640 10 monclient: _finish_auth 0 2024-03-25T18:16:17.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.956+0000 7f39de7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:16:17.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.956+0000 7f39de7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:17.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.956+0000 7f39efe93640 5 monclient: authenticate success, global_id 4131 2024-03-25T18:16:17.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.956+0000 7f39efe93640 10 monclient: _renew_subs 2024-03-25T18:16:17.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.956+0000 7f39efe93640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:16:17.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.956+0000 7f39efe93640 1 -- 172.21.15.98:0/2322273063 --> v1:172.21.15.97:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39e813a7d0 con 0x7f39e80b4700 2024-03-25T18:16:17.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.957+0000 7f39efe93640 10 client.4131.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:16:17.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.957+0000 7f39efe93640 10 monclient: _renew_subs 2024-03-25T18:16:17.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.957+0000 7f39efe93640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:16:17.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.957+0000 7f39efe93640 1 -- 172.21.15.98:0/2322273063 --> v1:172.21.15.97:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f39e81e3ce0 con 0x7f39e80b4700 2024-03-25T18:16:17.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.957+0000 7f39efe93640 1 librados: init done 2024-03-25T18:16:17.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.957+0000 7f39efe93640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:17.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.957+0000 7f39efe93640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:17.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.957+0000 7f39efe93640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:16:17.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.957+0000 7f39efe93640 1 -- 172.21.15.98:0/2322273063 --> v1:172.21.15.97:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39e81dbc60 con 0x7f39e80b4700 2024-03-25T18:16:17.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.961+0000 7f39de7fc640 1 -- 172.21.15.98:0/2322273063 <== mon.0 v1:172.21.15.97:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f39d8001c60 con 0x7f39e80b4700 2024-03-25T18:16:17.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.961+0000 7f39de7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:17.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.961+0000 7f39de7fc640 1 -- 172.21.15.98:0/2322273063 <== mon.0 v1:172.21.15.97:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f39d8004c30 con 0x7f39e80b4700 2024-03-25T18:16:17.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.961+0000 7f39de7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:17.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.961+0000 7f39de7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:16:17.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.961+0000 7f39de7fc640 10 monclient: dump: 2024-03-25T18:16:17.962 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-03-25T18:16:17.962 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:16:17.962 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:17.962 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:17.962 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:17.962 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-03-25T18:16:17.963 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:16:17.963 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:16:17.963 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:16:17.963 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-25T18:16:17.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.961+0000 7f39dffff640 4 set_mon_vals no callback set 2024-03-25T18:16:17.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.961+0000 7f39de7fc640 1 -- 172.21.15.98:0/2322273063 <== mon.0 v1:172.21.15.97:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f39d8005120 con 0x7f39e80b4700 2024-03-25T18:16:17.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.961+0000 7f39de7fc640 1 -- 172.21.15.98:0/2322273063 <== mon.0 v1:172.21.15.97:6789/0 8 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (unknown 2431733930 0 0) 0x7f39d8006640 con 0x7f39e80b4700 2024-03-25T18:16:17.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.961+0000 7f39de7fc640 10 client.4131.objecter ms_dispatch 0x7f39e8000b90 osd_map(4..4 src has 1..4) v4 2024-03-25T18:16:17.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.961+0000 7f39de7fc640 3 client.4131.objecter handle_osd_map got epochs [4,4] > 0 2024-03-25T18:16:17.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.961+0000 7f39de7fc640 3 client.4131.objecter handle_osd_map decoding full epoch 4 2024-03-25T18:16:17.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:17.961+0000 7f39de7fc640 20 client.4131.objecter dump_active .. 0 homeless 2024-03-25T18:16:17.986 INFO:tasks.ceph.mgr.y.smithi097.stderr:2024-03-25T18:16:17.985+0000 7f7cdf810040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-25T18:16:18.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.053+0000 7f39de7fc640 1 -- 172.21.15.98:0/2322273063 <== mon.0 v1:172.21.15.97:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f39d80062d0 con 0x7f39e80b4700 2024-03-25T18:16:18.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.053+0000 7f39de7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:18.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.053+0000 7f39de7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:16:18.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.125+0000 7f39efe93640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "136f1493-3b9c-4f43-a2e7-641482d69dae", "id": 2}] 2024-03-25T18:16:18.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.125+0000 7f39efe93640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "136f1493-3b9c-4f43-a2e7-641482d69dae", "id": 2}] 2024-03-25T18:16:18.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.126+0000 7f39efe93640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:16:18.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.126+0000 7f39efe93640 1 -- 172.21.15.98:0/2322273063 --> v1:172.21.15.97:6789/0 -- mon_command({"prefix": "osd new", "uuid": "136f1493-3b9c-4f43-a2e7-641482d69dae", "id": 2} v 0) v1 -- 0x7f39e804f730 con 0x7f39e80b4700 2024-03-25T18:16:18.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.186+0000 7f39de7fc640 1 -- 172.21.15.98:0/2322273063 <== mon.0 v1:172.21.15.97:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "136f1493-3b9c-4f43-a2e7-641482d69dae", "id": 2}]=0 v5) v1 ==== 112+0+2 (unknown 3443571027 0 216599125) 0x7f39d8006850 con 0x7f39e80b4700 2024-03-25T18:16:18.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.186+0000 7f39de7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "136f1493-3b9c-4f43-a2e7-641482d69dae", "id": 2}] 2024-03-25T18:16:18.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.186+0000 7f39de7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:16:18.188 INFO:teuthology.orchestra.run.smithi098.stdout:2 2024-03-25T18:16:18.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.187+0000 7f39efe93640 10 librados: watch_flush enter 2024-03-25T18:16:18.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.187+0000 7f39efe93640 10 librados: watch_flush exit 2024-03-25T18:16:18.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.187+0000 7f39efe93640 20 client.4131.objecter shutdown clearing up homeless session... 2024-03-25T18:16:18.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.187+0000 7f39efe93640 10 client.4131.objecter successfully canceled tick 2024-03-25T18:16:18.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.187+0000 7f39efe93640 10 monclient: shutdown 2024-03-25T18:16:18.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.187+0000 7f39efe93640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:18.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.187+0000 7f39efe93640 1 -- 172.21.15.98:0/2322273063 >> v1:172.21.15.97:6789/0 conn(0x7f39e80b4700 legacy=0x7f39e8139e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:18.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.187+0000 7f39efe93640 1 -- 172.21.15.98:0/2322273063 shutdown_connections 2024-03-25T18:16:18.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.187+0000 7f39efe93640 1 -- 172.21.15.98:0/2322273063 >> 172.21.15.98:0/2322273063 conn(0x7f39e80668e0 msgr2=0x7f39e813c2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:18.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.187+0000 7f39efe93640 1 -- 172.21.15.98:0/2322273063 shutdown_connections 2024-03-25T18:16:18.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.187+0000 7f39efe93640 1 -- 172.21.15.98:0/2322273063 wait complete. 2024-03-25T18:16:18.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.187+0000 7f39efe93640 1 librados: shutdown 2024-03-25T18:16:18.199 DEBUG:teuthology.orchestra.run.smithi098:> sudo ceph --cluster ceph osd new 397d538c-4f64-4daf-9e8b-662d09f91327 3 2024-03-25T18:16:18.314 INFO:tasks.ceph.mgr.x.smithi098.stderr:2024-03-25T18:16:18.313+0000 7f16671db040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-25T18:16:18.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.338+0000 7fb3ea425640 10 monclient: get_monmap_and_config 2024-03-25T18:16:18.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.338+0000 7fb3ea425640 10 monclient: build_initial_monmap 2024-03-25T18:16:18.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.338+0000 7fb3ea425640 10 monclient: monmap: 2024-03-25T18:16:18.339 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-03-25T18:16:18.339 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:16:18.339 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-03-25T18:16:18.339227+0000 2024-03-25T18:16:18.339 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-03-25T18:16:18.339227+0000 2024-03-25T18:16:18.339 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:18.339 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-03-25T18:16:18.339 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:16:18.339 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T18:16:18.339 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.97:6790/0 mon.noname-b 2024-03-25T18:16:18.339 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-25T18:16:18.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.338+0000 7fb3ea425640 1 Processor -- start 2024-03-25T18:16:18.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.339+0000 7fb3ea425640 1 -- start start 2024-03-25T18:16:18.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.339+0000 7fb3ea425640 10 monclient: init 2024-03-25T18:16:18.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.339+0000 7fb3ea425640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:18.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.339+0000 7fb3ea425640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:16:18.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.339+0000 7fb3ea425640 10 monclient(hunting): picked mon.noname-a con 0x7fb3e40b4700 addr v1:172.21.15.97:6789/0 2024-03-25T18:16:18.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.339+0000 7fb3ea425640 10 monclient(hunting): picked mon.noname-b con 0x7fb3e4142a20 addr v1:172.21.15.97:6790/0 2024-03-25T18:16:18.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.339+0000 7fb3ea425640 10 monclient(hunting): picked mon.noname-c con 0x7fb3e40a96b0 addr v1:172.21.15.98:6789/0 2024-03-25T18:16:18.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.339+0000 7fb3ea425640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3e40a9d60 con 0x7fb3e40b4700 2024-03-25T18:16:18.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.339+0000 7fb3ea425640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3e413ff10 con 0x7fb3e40a96b0 2024-03-25T18:16:18.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.339+0000 7fb3ea425640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3e40aa1f0 con 0x7fb3e4142a20 2024-03-25T18:16:18.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.339+0000 7fb3ea425640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:18.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.339+0000 7fb3ea425640 10 monclient(hunting): authenticate will time out at 7578.477496s 2024-03-25T18:16:18.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.340+0000 7fb3e37fe640 1 --1- >> v1:172.21.15.97:6790/0 conn(0x7fb3e4142a20 0x7fb3e413f0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6790/0 says I am v1:172.21.15.98:45570/0 (socket says 172.21.15.98:45570) 2024-03-25T18:16:18.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.340+0000 7fb3e37fe640 1 -- 172.21.15.98:0/1264875992 learned_addr learned my addr 172.21.15.98:0/1264875992 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-25T18:16:18.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.342+0000 7fb3e2ffd640 1 -- 172.21.15.98:0/1264875992 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3642537740 0 0) 0x7fb3e40aa1f0 con 0x7fb3e4142a20 2024-03-25T18:16:18.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.342+0000 7fb3e2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:16:18.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.342+0000 7fb3e2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:18.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.342+0000 7fb3e2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:18.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.342+0000 7fb3e2ffd640 10 monclient(hunting): my global_id is 4124 2024-03-25T18:16:18.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.342+0000 7fb3e2ffd640 1 -- 172.21.15.98:0/1264875992 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3cc003520 con 0x7fb3e4142a20 2024-03-25T18:16:18.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.342+0000 7fb3e2ffd640 1 -- 172.21.15.98:0/1264875992 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 525905781 0 0) 0x7fb3e413ff10 con 0x7fb3e40a96b0 2024-03-25T18:16:18.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.342+0000 7fb3e2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:16:18.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.342+0000 7fb3e2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:18.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.342+0000 7fb3e2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:18.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.342+0000 7fb3e2ffd640 10 monclient(hunting): my global_id is 4123 2024-03-25T18:16:18.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.342+0000 7fb3e2ffd640 1 -- 172.21.15.98:0/1264875992 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3e40aa1f0 con 0x7fb3e40a96b0 2024-03-25T18:16:18.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.342+0000 7fb3e2ffd640 1 -- 172.21.15.98:0/1264875992 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 161348314 0 0) 0x7fb3e40a9d60 con 0x7fb3e40b4700 2024-03-25T18:16:18.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.342+0000 7fb3e2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:16:18.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.342+0000 7fb3e2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:18.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.342+0000 7fb3e2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:18.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.342+0000 7fb3e2ffd640 10 monclient(hunting): my global_id is 4134 2024-03-25T18:16:18.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.342+0000 7fb3e2ffd640 1 -- 172.21.15.98:0/1264875992 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3e413ff10 con 0x7fb3e40b4700 2024-03-25T18:16:18.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.345+0000 7fb3e2ffd640 1 -- 172.21.15.98:0/1264875992 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2067097629 0 0) 0x7fb3e40aa1f0 con 0x7fb3e40a96b0 2024-03-25T18:16:18.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.345+0000 7fb3e2ffd640 1 -- 172.21.15.98:0/1264875992 >> v1:172.21.15.97:6790/0 conn(0x7fb3e4142a20 legacy=0x7fb3e413f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:18.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.345+0000 7fb3e2ffd640 1 -- 172.21.15.98:0/1264875992 >> v1:172.21.15.97:6789/0 conn(0x7fb3e40b4700 legacy=0x7fb3e4142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:18.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.345+0000 7fb3e2ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:16:18.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.345+0000 7fb3e2ffd640 1 monclient: found mon.noname-c 2024-03-25T18:16:18.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.345+0000 7fb3e2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:18.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.345+0000 7fb3e2ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.98:6789/0 2024-03-25T18:16:18.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.345+0000 7fb3e2ffd640 1 -- 172.21.15.98:0/1264875992 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3e40b4ae0 con 0x7fb3e40a96b0 2024-03-25T18:16:18.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.345+0000 7fb3e2ffd640 10 monclient: _finish_auth 0 2024-03-25T18:16:18.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.345+0000 7fb3e2ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:16:18.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.345+0000 7fb3e2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:18.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.345+0000 7fb3e2ffd640 1 -- 172.21.15.98:0/1264875992 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7fb3dc0019f0 con 0x7fb3e40a96b0 2024-03-25T18:16:18.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.345+0000 7fb3ea425640 5 monclient: authenticate success, global_id 4123 2024-03-25T18:16:18.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.345+0000 7fb3ea425640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:18.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.345+0000 7fb3e2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:18.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.345+0000 7fb3e2ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:16:18.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.345+0000 7fb3e2ffd640 10 monclient: dump: 2024-03-25T18:16:18.348 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-03-25T18:16:18.348 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:16:18.348 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:18.348 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:18.348 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:18.348 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-03-25T18:16:18.348 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:16:18.348 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:16:18.348 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:16:18.348 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-25T18:16:18.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.345+0000 7fb3ea425640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:18.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.348+0000 7fb3e2ffd640 1 -- 172.21.15.98:0/1264875992 <== mon.1 v1:172.21.15.98:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb3dc001b50 con 0x7fb3e40a96b0 2024-03-25T18:16:18.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.348+0000 7fb3e2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:18.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.348+0000 7fb3e2ffd640 1 -- 172.21.15.98:0/1264875992 <== mon.1 v1:172.21.15.98:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7fb3dc001f60 con 0x7fb3e40a96b0 2024-03-25T18:16:18.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.348+0000 7fb3ea425640 10 monclient: get_monmap_and_config success 2024-03-25T18:16:18.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.348+0000 7fb3ea425640 4 set_mon_vals no callback set 2024-03-25T18:16:18.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.348+0000 7fb3e2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:18.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.348+0000 7fb3e2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:16:18.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.348+0000 7fb3e2ffd640 10 monclient: dump: 2024-03-25T18:16:18.350 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-03-25T18:16:18.350 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:16:18.351 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:18.351 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:18.351 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:18.351 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-03-25T18:16:18.351 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:16:18.351 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:16:18.351 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:16:18.351 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-25T18:16:18.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.348+0000 7fb3ea425640 10 monclient: shutdown 2024-03-25T18:16:18.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.348+0000 7fb3ea425640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:18.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.348+0000 7fb3ea425640 1 -- 172.21.15.98:0/1264875992 >> v1:172.21.15.98:6789/0 conn(0x7fb3e40a96b0 legacy=0x7fb3e413f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:18.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.348+0000 7fb3ea425640 1 -- 172.21.15.98:0/1264875992 shutdown_connections 2024-03-25T18:16:18.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.348+0000 7fb3ea425640 1 -- 172.21.15.98:0/1264875992 >> 172.21.15.98:0/1264875992 conn(0x7fb3e40668e0 msgr2=0x7fb3e4066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:18.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.348+0000 7fb3ea425640 1 -- 172.21.15.98:0/1264875992 shutdown_connections 2024-03-25T18:16:18.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.348+0000 7fb3ea425640 1 -- 172.21.15.98:0/1264875992 wait complete. 2024-03-25T18:16:18.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.349+0000 7fb3ea425640 10 monclient: build_initial_monmap 2024-03-25T18:16:18.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.349+0000 7fb3ea425640 10 monclient: monmap: 2024-03-25T18:16:18.352 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-03-25T18:16:18.352 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:16:18.352 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-03-25T18:16:18.352 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-03-25T18:16:18.352 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:18.352 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-03-25T18:16:18.352 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:16:18.352 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T18:16:18.352 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.97:6790/0 mon.noname-c 2024-03-25T18:16:18.352 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-25T18:16:18.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.349+0000 7fb3ea425640 1 librados: starting msgr at 2024-03-25T18:16:18.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.349+0000 7fb3ea425640 1 librados: starting objecter 2024-03-25T18:16:18.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.350+0000 7fb3ea425640 1 Processor -- start 2024-03-25T18:16:18.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.350+0000 7fb3ea425640 1 -- start start 2024-03-25T18:16:18.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.350+0000 7fb3ea425640 1 librados: setting wanted keys 2024-03-25T18:16:18.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.350+0000 7fb3ea425640 1 librados: calling monclient init 2024-03-25T18:16:18.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.350+0000 7fb3ea425640 10 monclient: init 2024-03-25T18:16:18.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.350+0000 7fb3ea425640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:18.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.350+0000 7fb3ea425640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:16:18.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.350+0000 7fb3ea425640 10 monclient(hunting): picked mon.noname-c con 0x7fb3e40a96b0 addr v1:172.21.15.97:6790/0 2024-03-25T18:16:18.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.350+0000 7fb3ea425640 10 monclient(hunting): picked mon.noname-a con 0x7fb3e40b4700 addr v1:172.21.15.97:6789/0 2024-03-25T18:16:18.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.350+0000 7fb3ea425640 10 monclient(hunting): picked mon.noname-b con 0x7fb3e4142a20 addr v1:172.21.15.98:6789/0 2024-03-25T18:16:18.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.350+0000 7fb3ea425640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3e4138430 con 0x7fb3e40b4700 2024-03-25T18:16:18.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.350+0000 7fb3ea425640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3e41d3fd0 con 0x7fb3e4142a20 2024-03-25T18:16:18.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.350+0000 7fb3ea425640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3e41d4f30 con 0x7fb3e40a96b0 2024-03-25T18:16:18.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.350+0000 7fb3ea425640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:18.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.350+0000 7fb3ea425640 10 monclient(hunting): authenticate will time out at 7578.488786s 2024-03-25T18:16:18.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.351+0000 7fb3e3fff640 1 --1- >> v1:172.21.15.97:6790/0 conn(0x7fb3e40a96b0 0x7fb3e4137d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.97:6790/0 says I am v1:172.21.15.98:45572/0 (socket says 172.21.15.98:45572) 2024-03-25T18:16:18.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.351+0000 7fb3e3fff640 1 -- 172.21.15.98:0/1918664456 learned_addr learned my addr 172.21.15.98:0/1918664456 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-25T18:16:18.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.351+0000 7fb3e0ff9640 10 client.?.objecter ms_handle_connect 0x7fb3e40a96b0 2024-03-25T18:16:18.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.351+0000 7fb3e0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:18.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.351+0000 7fb3e0ff9640 10 client.?.objecter ms_handle_connect 0x7fb3e4142a20 2024-03-25T18:16:18.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.351+0000 7fb3e0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:18.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.351+0000 7fb3e0ff9640 10 client.?.objecter ms_handle_connect 0x7fb3e40b4700 2024-03-25T18:16:18.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.351+0000 7fb3e0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:18.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.353+0000 7fb3e0ff9640 1 -- 172.21.15.98:0/1918664456 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 573233143 0 0) 0x7fb3e41d3fd0 con 0x7fb3e4142a20 2024-03-25T18:16:18.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.353+0000 7fb3e0ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:16:18.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.353+0000 7fb3e0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:18.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.353+0000 7fb3e0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:18.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.353+0000 7fb3e0ff9640 10 monclient(hunting): my global_id is 4126 2024-03-25T18:16:18.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.353+0000 7fb3e0ff9640 1 -- 172.21.15.98:0/1918664456 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3d0004eb0 con 0x7fb3e4142a20 2024-03-25T18:16:18.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.353+0000 7fb3e0ff9640 1 -- 172.21.15.98:0/1918664456 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3425522115 0 0) 0x7fb3e41d4f30 con 0x7fb3e40a96b0 2024-03-25T18:16:18.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.353+0000 7fb3e0ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:16:18.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.353+0000 7fb3e0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:18.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.353+0000 7fb3e0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:18.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.353+0000 7fb3e0ff9640 10 monclient(hunting): my global_id is 4127 2024-03-25T18:16:18.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.353+0000 7fb3e0ff9640 1 -- 172.21.15.98:0/1918664456 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3e41d3fd0 con 0x7fb3e40a96b0 2024-03-25T18:16:18.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.354+0000 7fb3e0ff9640 1 -- 172.21.15.98:0/1918664456 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2844001822 0 0) 0x7fb3e4138430 con 0x7fb3e40b4700 2024-03-25T18:16:18.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.354+0000 7fb3e0ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:16:18.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.354+0000 7fb3e0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:18.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.354+0000 7fb3e0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:18.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.354+0000 7fb3e0ff9640 10 monclient(hunting): my global_id is 4137 2024-03-25T18:16:18.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.354+0000 7fb3e0ff9640 1 -- 172.21.15.98:0/1918664456 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3e41d4f30 con 0x7fb3e40b4700 2024-03-25T18:16:18.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.358+0000 7fb3e0ff9640 1 -- 172.21.15.98:0/1918664456 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 696590674 0 0) 0x7fb3d0004eb0 con 0x7fb3e4142a20 2024-03-25T18:16:18.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.358+0000 7fb3e0ff9640 1 -- 172.21.15.98:0/1918664456 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb3e4138430 con 0x7fb3e4142a20 2024-03-25T18:16:18.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.358+0000 7fb3e0ff9640 1 -- 172.21.15.98:0/1918664456 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7fb3dc001e60 con 0x7fb3e4142a20 2024-03-25T18:16:18.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.358+0000 7fb3e0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:18.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.358+0000 7fb3e0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:16:18.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.358+0000 7fb3e0ff9640 10 monclient(hunting): dump: 2024-03-25T18:16:18.360 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-03-25T18:16:18.360 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:16:18.360 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:18.360 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:18.360 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:18.360 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-03-25T18:16:18.360 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:16:18.360 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:16:18.360 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:16:18.360 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-25T18:16:18.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.358+0000 7fb3e0ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:16:18.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.358+0000 7fb3e0ff9640 1 -- 172.21.15.98:0/1918664456 <== mon.2 v1:172.21.15.97:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2923102774 0 0) 0x7fb3e41d3fd0 con 0x7fb3e40a96b0 2024-03-25T18:16:18.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.358+0000 7fb3e0ff9640 1 -- 172.21.15.98:0/1918664456 --> v1:172.21.15.97:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb3d0004eb0 con 0x7fb3e40a96b0 2024-03-25T18:16:18.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.358+0000 7fb3e0ff9640 1 -- 172.21.15.98:0/1918664456 <== mon.2 v1:172.21.15.97:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7fb3d40019f0 con 0x7fb3e40a96b0 2024-03-25T18:16:18.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.358+0000 7fb3e0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:18.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.358+0000 7fb3e0ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:16:18.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.358+0000 7fb3e0ff9640 10 monclient(hunting): dump: 2024-03-25T18:16:18.361 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-03-25T18:16:18.361 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:16:18.361 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:18.361 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:18.361 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:18.361 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-03-25T18:16:18.361 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:16:18.361 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:16:18.361 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:16:18.361 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-25T18:16:18.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.358+0000 7fb3e0ff9640 1 -- 172.21.15.98:0/1918664456 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2668159241 0 0) 0x7fb3e41d4f30 con 0x7fb3e40b4700 2024-03-25T18:16:18.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.358+0000 7fb3e0ff9640 1 -- 172.21.15.98:0/1918664456 --> v1:172.21.15.97:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb3e41d3fd0 con 0x7fb3e40b4700 2024-03-25T18:16:18.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.359+0000 7fb3e0ff9640 1 -- 172.21.15.98:0/1918664456 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7fb3d80017d0 con 0x7fb3e40b4700 2024-03-25T18:16:18.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.359+0000 7fb3e0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:18.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.359+0000 7fb3e0ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:16:18.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.359+0000 7fb3e0ff9640 10 monclient(hunting): dump: 2024-03-25T18:16:18.363 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-03-25T18:16:18.363 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:16:18.363 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:18.363 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:18.363 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:18.364 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-03-25T18:16:18.364 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:16:18.364 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:16:18.364 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:16:18.364 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-25T18:16:18.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.361+0000 7fb3e0ff9640 1 -- 172.21.15.98:0/1918664456 <== mon.2 v1:172.21.15.97:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3819619536 0 0) 0x7fb3d0004eb0 con 0x7fb3e40a96b0 2024-03-25T18:16:18.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.361+0000 7fb3e0ff9640 1 -- 172.21.15.98:0/1918664456 >> v1:172.21.15.98:6789/0 conn(0x7fb3e4142a20 legacy=0x7fb3e412ff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:18.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.361+0000 7fb3e0ff9640 1 -- 172.21.15.98:0/1918664456 >> v1:172.21.15.97:6789/0 conn(0x7fb3e40b4700 legacy=0x7fb3e41d4820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:18.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.361+0000 7fb3e0ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:16:18.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.361+0000 7fb3e0ff9640 1 monclient: found mon.c 2024-03-25T18:16:18.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.361+0000 7fb3e0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:18.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.361+0000 7fb3e0ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.97:6790/0 2024-03-25T18:16:18.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.361+0000 7fb3e0ff9640 1 -- 172.21.15.98:0/1918664456 --> v1:172.21.15.97:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3e4134380 con 0x7fb3e40a96b0 2024-03-25T18:16:18.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.361+0000 7fb3e0ff9640 10 monclient: _finish_auth 0 2024-03-25T18:16:18.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.361+0000 7fb3e0ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:16:18.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.361+0000 7fb3e0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:18.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.361+0000 7fb3ea425640 5 monclient: authenticate success, global_id 4127 2024-03-25T18:16:18.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.361+0000 7fb3ea425640 10 monclient: _renew_subs 2024-03-25T18:16:18.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.361+0000 7fb3ea425640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.97:6790/0 2024-03-25T18:16:18.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.361+0000 7fb3ea425640 1 -- 172.21.15.98:0/1918664456 --> v1:172.21.15.97:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3e4134630 con 0x7fb3e40a96b0 2024-03-25T18:16:18.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.361+0000 7fb3ea425640 10 client.4127.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:16:18.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.361+0000 7fb3ea425640 10 monclient: _renew_subs 2024-03-25T18:16:18.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.361+0000 7fb3ea425640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.97:6790/0 2024-03-25T18:16:18.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.361+0000 7fb3ea425640 1 -- 172.21.15.98:0/1918664456 --> v1:172.21.15.97:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3e4137660 con 0x7fb3e40a96b0 2024-03-25T18:16:18.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.361+0000 7fb3ea425640 1 librados: init done 2024-03-25T18:16:18.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.362+0000 7fb3ea425640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:18.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.362+0000 7fb3ea425640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:18.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.362+0000 7fb3ea425640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.97:6790/0 2024-03-25T18:16:18.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.362+0000 7fb3ea425640 1 -- 172.21.15.98:0/1918664456 --> v1:172.21.15.97:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3e41d4670 con 0x7fb3e40a96b0 2024-03-25T18:16:18.385 INFO:tasks.ceph.mgr.y.smithi097.stderr:2024-03-25T18:16:18.384+0000 7f7cdf810040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-25T18:16:18.452 INFO:tasks.ceph.mgr.x.smithi098.stderr:2024-03-25T18:16:18.452+0000 7f16671db040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-25T18:16:18.525 INFO:tasks.ceph.mgr.y.smithi097.stderr:2024-03-25T18:16:18.524+0000 7f7cdf810040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-25T18:16:18.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.554+0000 7fb3e0ff9640 1 -- 172.21.15.98:0/1918664456 <== mon.2 v1:172.21.15.97:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb3d4003610 con 0x7fb3e40a96b0 2024-03-25T18:16:18.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.555+0000 7fb3e0ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:18.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.555+0000 7fb3e0ff9640 1 -- 172.21.15.98:0/1918664456 <== mon.2 v1:172.21.15.97:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7fb3d40039a0 con 0x7fb3e40a96b0 2024-03-25T18:16:18.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.555+0000 7fb3e0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:18.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.555+0000 7fb3e0ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:16:18.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.555+0000 7fb3e0ff9640 10 monclient: dump: 2024-03-25T18:16:18.556 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-03-25T18:16:18.556 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:16:18.556 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:18.556 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:18.556 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:18.556 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-03-25T18:16:18.556 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:16:18.556 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:16:18.556 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:16:18.556 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-25T18:16:18.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.555+0000 7fb3e0ff9640 1 -- 172.21.15.98:0/1918664456 <== mon.2 v1:172.21.15.97:6790/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fb3d4003f50 con 0x7fb3e40a96b0 2024-03-25T18:16:18.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.555+0000 7fb3e27fc640 4 set_mon_vals no callback set 2024-03-25T18:16:18.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.558+0000 7fb3e0ff9640 1 -- 172.21.15.98:0/1918664456 <== mon.2 v1:172.21.15.97:6790/0 8 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (unknown 3990004627 0 0) 0x7fb3d40066b0 con 0x7fb3e40a96b0 2024-03-25T18:16:18.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.558+0000 7fb3e0ff9640 10 client.4127.objecter ms_dispatch 0x7fb3e4000b90 osd_map(5..5 src has 1..5) v4 2024-03-25T18:16:18.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.558+0000 7fb3e0ff9640 3 client.4127.objecter handle_osd_map got epochs [5,5] > 0 2024-03-25T18:16:18.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.558+0000 7fb3e0ff9640 3 client.4127.objecter handle_osd_map decoding full epoch 5 2024-03-25T18:16:18.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.558+0000 7fb3e0ff9640 20 client.4127.objecter dump_active .. 0 homeless 2024-03-25T18:16:18.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.558+0000 7fb3e0ff9640 1 -- 172.21.15.98:0/1918664456 <== mon.2 v1:172.21.15.97:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fb3d4004480 con 0x7fb3e40a96b0 2024-03-25T18:16:18.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.558+0000 7fb3e0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:18.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.558+0000 7fb3e0ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:16:18.571 INFO:tasks.ceph.mgr.x.smithi098.stderr:2024-03-25T18:16:18.570+0000 7f16671db040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-25T18:16:18.618 INFO:tasks.ceph.mgr.x.smithi098.stderr:2024-03-25T18:16:18.617+0000 7f16671db040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-25T18:16:18.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.633+0000 7fb3ea425640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "397d538c-4f64-4daf-9e8b-662d09f91327", "id": 3}] 2024-03-25T18:16:18.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.633+0000 7fb3ea425640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "397d538c-4f64-4daf-9e8b-662d09f91327", "id": 3}] 2024-03-25T18:16:18.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.633+0000 7fb3ea425640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.97:6790/0 2024-03-25T18:16:18.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.633+0000 7fb3ea425640 1 -- 172.21.15.98:0/1918664456 --> v1:172.21.15.97:6790/0 -- mon_command({"prefix": "osd new", "uuid": "397d538c-4f64-4daf-9e8b-662d09f91327", "id": 3} v 0) v1 -- 0x7fb3e4139410 con 0x7fb3e40a96b0 2024-03-25T18:16:18.644 INFO:tasks.ceph.mgr.y.smithi097.stderr:2024-03-25T18:16:18.643+0000 7f7cdf810040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-25T18:16:18.692 INFO:tasks.ceph.mgr.y.smithi097.stderr:2024-03-25T18:16:18.691+0000 7f7cdf810040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-25T18:16:18.752 INFO:tasks.ceph.mgr.x.smithi098.stderr:2024-03-25T18:16:18.751+0000 7f16671db040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-25T18:16:18.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.766+0000 7fb3e0ff9640 1 -- 172.21.15.98:0/1918664456 <== mon.2 v1:172.21.15.97:6790/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "397d538c-4f64-4daf-9e8b-662d09f91327", "id": 3}]=0 v6) v1 ==== 112+0+2 (unknown 994293433 0 525046306) 0x7fb3e4139410 con 0x7fb3e40a96b0 2024-03-25T18:16:18.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.766+0000 7fb3e0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "397d538c-4f64-4daf-9e8b-662d09f91327", "id": 3}] 2024-03-25T18:16:18.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.766+0000 7fb3e0ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:16:18.767 INFO:teuthology.orchestra.run.smithi098.stdout:3 2024-03-25T18:16:18.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.767+0000 7fb3ea425640 10 librados: watch_flush enter 2024-03-25T18:16:18.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.767+0000 7fb3ea425640 10 librados: watch_flush exit 2024-03-25T18:16:18.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.767+0000 7fb3ea425640 20 client.4127.objecter shutdown clearing up homeless session... 2024-03-25T18:16:18.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.767+0000 7fb3ea425640 10 client.4127.objecter successfully canceled tick 2024-03-25T18:16:18.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.767+0000 7fb3ea425640 10 monclient: shutdown 2024-03-25T18:16:18.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.767+0000 7fb3ea425640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:18.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.767+0000 7fb3ea425640 1 -- 172.21.15.98:0/1918664456 >> v1:172.21.15.97:6790/0 conn(0x7fb3e40a96b0 legacy=0x7fb3e4137d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:18.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.767+0000 7fb3ea425640 1 -- 172.21.15.98:0/1918664456 shutdown_connections 2024-03-25T18:16:18.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.767+0000 7fb3ea425640 1 -- 172.21.15.98:0/1918664456 >> 172.21.15.98:0/1918664456 conn(0x7fb3e40668e0 msgr2=0x7fb3e40b5a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:18.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.767+0000 7fb3ea425640 1 -- 172.21.15.98:0/1918664456 shutdown_connections 2024-03-25T18:16:18.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.767+0000 7fb3ea425640 1 -- 172.21.15.98:0/1918664456 wait complete. 2024-03-25T18:16:18.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.767+0000 7fb3ea425640 1 librados: shutdown 2024-03-25T18:16:18.778 DEBUG:teuthology.orchestra.run.smithi098:> sudo ceph --cluster ceph osd new f4f1df08-663f-4bda-93f8-3878ab7c24b3 4 2024-03-25T18:16:18.809 INFO:tasks.ceph.mgr.y.smithi097.stderr:2024-03-25T18:16:18.808+0000 7f7cdf810040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-25T18:16:18.853 INFO:tasks.ceph.mgr.x.smithi098.stderr:2024-03-25T18:16:18.852+0000 7f16671db040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-25T18:16:18.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.877+0000 7f2cf8855640 10 monclient: get_monmap_and_config 2024-03-25T18:16:18.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.877+0000 7f2cf8855640 10 monclient: build_initial_monmap 2024-03-25T18:16:18.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.877+0000 7f2cf8855640 10 monclient: monmap: 2024-03-25T18:16:18.878 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-03-25T18:16:18.878 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:16:18.878 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-03-25T18:16:18.878532+0000 2024-03-25T18:16:18.878 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-03-25T18:16:18.878532+0000 2024-03-25T18:16:18.878 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:18.878 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-03-25T18:16:18.878 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:16:18.879 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T18:16:18.879 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.97:6790/0 mon.noname-b 2024-03-25T18:16:18.879 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-25T18:16:18.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.878+0000 7f2cf8855640 1 Processor -- start 2024-03-25T18:16:18.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.878+0000 7f2cf8855640 1 -- start start 2024-03-25T18:16:18.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.878+0000 7f2cf8855640 10 monclient: init 2024-03-25T18:16:18.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.878+0000 7f2cf8855640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:18.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.878+0000 7f2cf8855640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:16:18.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.878+0000 7f2cf8855640 10 monclient(hunting): picked mon.noname-c con 0x7f2cf40a9920 addr v1:172.21.15.98:6789/0 2024-03-25T18:16:18.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.878+0000 7f2cf8855640 10 monclient(hunting): picked mon.noname-a con 0x7f2cf40b45a0 addr v1:172.21.15.97:6789/0 2024-03-25T18:16:18.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.878+0000 7f2cf8855640 10 monclient(hunting): picked mon.noname-b con 0x7f2cf4146450 addr v1:172.21.15.97:6790/0 2024-03-25T18:16:18.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.878+0000 7f2cf8855640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cf406b100 con 0x7f2cf40b45a0 2024-03-25T18:16:18.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.878+0000 7f2cf8855640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cf40b2940 con 0x7f2cf40a9920 2024-03-25T18:16:18.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.878+0000 7f2cf8855640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cf413fec0 con 0x7f2cf4146450 2024-03-25T18:16:18.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.878+0000 7f2cf8855640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:18.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.878+0000 7f2cf8855640 10 monclient(hunting): authenticate will time out at 7579.016905s 2024-03-25T18:16:18.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.880+0000 7f2cf259c640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f2cf40a9920 0x7f2cf413f7b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:60700/0 (socket says 172.21.15.98:60700) 2024-03-25T18:16:18.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.880+0000 7f2cf259c640 1 -- 172.21.15.98:0/359896814 learned_addr learned my addr 172.21.15.98:0/359896814 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-25T18:16:18.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.893+0000 7f2cf159a640 1 -- 172.21.15.98:0/359896814 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1677141423 0 0) 0x7f2cf406b100 con 0x7f2cf40b45a0 2024-03-25T18:16:18.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.894+0000 7f2cf159a640 10 monclient(hunting): _negotiate 2024-03-25T18:16:18.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.894+0000 7f2cf159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:18.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.894+0000 7f2cf159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:18.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.894+0000 7f2cf159a640 10 monclient(hunting): my global_id is 4140 2024-03-25T18:16:18.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.894+0000 7f2cf159a640 1 -- 172.21.15.98:0/359896814 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2cdc003520 con 0x7f2cf40b45a0 2024-03-25T18:16:18.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.896+0000 7f2cf159a640 1 -- 172.21.15.98:0/359896814 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2331639805 0 0) 0x7f2cf413fec0 con 0x7f2cf4146450 2024-03-25T18:16:18.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.896+0000 7f2cf159a640 10 monclient(hunting): _negotiate 2024-03-25T18:16:18.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.896+0000 7f2cf159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:18.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.896+0000 7f2cf159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:18.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.896+0000 7f2cf159a640 10 monclient(hunting): my global_id is 4130 2024-03-25T18:16:18.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.896+0000 7f2cf159a640 1 -- 172.21.15.98:0/359896814 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2cf406b100 con 0x7f2cf4146450 2024-03-25T18:16:18.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.898+0000 7f2cf159a640 1 -- 172.21.15.98:0/359896814 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2110778832 0 0) 0x7f2cf40b2940 con 0x7f2cf40a9920 2024-03-25T18:16:18.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.898+0000 7f2cf159a640 10 monclient(hunting): _negotiate 2024-03-25T18:16:18.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.898+0000 7f2cf159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:18.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.898+0000 7f2cf159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:18.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.898+0000 7f2cf159a640 10 monclient(hunting): my global_id is 4129 2024-03-25T18:16:18.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.898+0000 7f2cf159a640 1 -- 172.21.15.98:0/359896814 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2cf413fec0 con 0x7f2cf40a9920 2024-03-25T18:16:18.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.898+0000 7f2cf159a640 1 -- 172.21.15.98:0/359896814 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1544432675 0 0) 0x7f2cdc003520 con 0x7f2cf40b45a0 2024-03-25T18:16:18.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.898+0000 7f2cf159a640 1 -- 172.21.15.98:0/359896814 >> v1:172.21.15.97:6790/0 conn(0x7f2cf4146450 legacy=0x7f2cf414dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:18.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.898+0000 7f2cf159a640 1 -- 172.21.15.98:0/359896814 >> v1:172.21.15.98:6789/0 conn(0x7f2cf40a9920 legacy=0x7f2cf413f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:18.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.898+0000 7f2cf159a640 10 monclient: _finish_hunting 0 2024-03-25T18:16:18.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.898+0000 7f2cf159a640 1 monclient: found mon.noname-a 2024-03-25T18:16:18.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.898+0000 7f2cf159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:18.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.898+0000 7f2cf159a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.97:6789/0 2024-03-25T18:16:18.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.898+0000 7f2cf159a640 1 -- 172.21.15.98:0/359896814 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cf4140050 con 0x7f2cf40b45a0 2024-03-25T18:16:18.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.898+0000 7f2cf159a640 10 monclient: _finish_auth 0 2024-03-25T18:16:18.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.898+0000 7f2cf159a640 10 monclient: _check_auth_tickets 2024-03-25T18:16:18.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.898+0000 7f2cf159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:18.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.898+0000 7f2cf159a640 1 -- 172.21.15.98:0/359896814 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f2ce8001690 con 0x7f2cf40b45a0 2024-03-25T18:16:18.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.898+0000 7f2cf8855640 5 monclient: authenticate success, global_id 4140 2024-03-25T18:16:18.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.898+0000 7f2cf8855640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:18.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.898+0000 7f2cf159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:18.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.898+0000 7f2cf159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:16:18.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.898+0000 7f2cf159a640 10 monclient: dump: 2024-03-25T18:16:18.901 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-03-25T18:16:18.901 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:16:18.901 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:18.901 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:18.902 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:18.902 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-03-25T18:16:18.902 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:16:18.902 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:16:18.902 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:16:18.902 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-25T18:16:18.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.898+0000 7f2cf8855640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:18.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.901+0000 7f2cf159a640 1 -- 172.21.15.98:0/359896814 <== mon.0 v1:172.21.15.97:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2ce80017f0 con 0x7f2cf40b45a0 2024-03-25T18:16:18.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.901+0000 7f2cf159a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:18.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.901+0000 7f2cf159a640 1 -- 172.21.15.98:0/359896814 <== mon.0 v1:172.21.15.97:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f2ce8001a60 con 0x7f2cf40b45a0 2024-03-25T18:16:18.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.901+0000 7f2cf159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:18.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.901+0000 7f2cf159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:16:18.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.901+0000 7f2cf159a640 10 monclient: dump: 2024-03-25T18:16:18.904 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-03-25T18:16:18.904 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:16:18.904 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:18.904 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:18.904 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:18.904 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-03-25T18:16:18.904 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:16:18.904 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:16:18.904 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:16:18.904 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-25T18:16:18.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.901+0000 7f2cf8855640 10 monclient: get_monmap_and_config success 2024-03-25T18:16:18.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.901+0000 7f2cf8855640 4 set_mon_vals no callback set 2024-03-25T18:16:18.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.902+0000 7f2cf8855640 10 monclient: shutdown 2024-03-25T18:16:18.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.902+0000 7f2cf8855640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:18.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.902+0000 7f2cf8855640 1 -- 172.21.15.98:0/359896814 >> v1:172.21.15.97:6789/0 conn(0x7f2cf40b45a0 legacy=0x7f2cf4145d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:18.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.902+0000 7f2cf8855640 1 -- 172.21.15.98:0/359896814 shutdown_connections 2024-03-25T18:16:18.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.902+0000 7f2cf8855640 1 -- 172.21.15.98:0/359896814 >> 172.21.15.98:0/359896814 conn(0x7f2cf40b7f40 msgr2=0x7f2cf40b8350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:18.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.902+0000 7f2cf8855640 1 -- 172.21.15.98:0/359896814 shutdown_connections 2024-03-25T18:16:18.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.902+0000 7f2cf8855640 1 -- 172.21.15.98:0/359896814 wait complete. 2024-03-25T18:16:18.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.903+0000 7f2cf8855640 10 monclient: build_initial_monmap 2024-03-25T18:16:18.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.903+0000 7f2cf8855640 10 monclient: monmap: 2024-03-25T18:16:18.905 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-03-25T18:16:18.905 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:16:18.905 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-03-25T18:16:18.905 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-03-25T18:16:18.905 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:18.905 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-03-25T18:16:18.905 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:16:18.906 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T18:16:18.906 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.97:6790/0 mon.noname-c 2024-03-25T18:16:18.906 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-25T18:16:18.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.903+0000 7f2cf8855640 1 librados: starting msgr at 2024-03-25T18:16:18.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.903+0000 7f2cf8855640 1 librados: starting objecter 2024-03-25T18:16:18.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.903+0000 7f2cf8855640 1 Processor -- start 2024-03-25T18:16:18.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.903+0000 7f2cf8855640 1 -- start start 2024-03-25T18:16:18.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.903+0000 7f2cf8855640 1 librados: setting wanted keys 2024-03-25T18:16:18.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.903+0000 7f2cf8855640 1 librados: calling monclient init 2024-03-25T18:16:18.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.903+0000 7f2cf8855640 10 monclient: init 2024-03-25T18:16:18.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.904+0000 7f2cf8855640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:18.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.904+0000 7f2cf8855640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:16:18.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.904+0000 7f2cf8855640 10 monclient(hunting): picked mon.noname-c con 0x7f2cf40a9920 addr v1:172.21.15.97:6790/0 2024-03-25T18:16:18.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.904+0000 7f2cf8855640 10 monclient(hunting): picked mon.noname-a con 0x7f2cf40b45a0 addr v1:172.21.15.97:6789/0 2024-03-25T18:16:18.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.904+0000 7f2cf8855640 10 monclient(hunting): picked mon.noname-b con 0x7f2cf4146450 addr v1:172.21.15.98:6789/0 2024-03-25T18:16:18.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.904+0000 7f2cf8855640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cf4144610 con 0x7f2cf40b45a0 2024-03-25T18:16:18.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.904+0000 7f2cf8855640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cf41453d0 con 0x7f2cf4146450 2024-03-25T18:16:18.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.904+0000 7f2cf8855640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cf41dab70 con 0x7f2cf40a9920 2024-03-25T18:16:18.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.904+0000 7f2cf8855640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:18.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.904+0000 7f2cf8855640 10 monclient(hunting): authenticate will time out at 7579.042067s 2024-03-25T18:16:18.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.904+0000 7f2cf259c640 1 --1- >> v1:172.21.15.97:6790/0 conn(0x7f2cf40a9920 0x7f2cf4142830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.97:6790/0 says I am v1:172.21.15.98:45594/0 (socket says 172.21.15.98:45594) 2024-03-25T18:16:18.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.904+0000 7f2cf259c640 1 -- 172.21.15.98:0/738370650 learned_addr learned my addr 172.21.15.98:0/738370650 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-25T18:16:18.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.904+0000 7f2ce2ffd640 10 client.?.objecter ms_handle_connect 0x7f2cf4146450 2024-03-25T18:16:18.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.904+0000 7f2ce2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:18.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.904+0000 7f2ce2ffd640 10 client.?.objecter ms_handle_connect 0x7f2cf40a9920 2024-03-25T18:16:18.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.904+0000 7f2ce2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:18.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.905+0000 7f2ce2ffd640 10 client.?.objecter ms_handle_connect 0x7f2cf40b45a0 2024-03-25T18:16:18.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.905+0000 7f2ce2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:18.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.907+0000 7f2ce2ffd640 1 -- 172.21.15.98:0/738370650 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3227600858 0 0) 0x7f2cf41453d0 con 0x7f2cf4146450 2024-03-25T18:16:18.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.907+0000 7f2ce2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:16:18.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.907+0000 7f2ce2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:18.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.907+0000 7f2ce2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:18.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.907+0000 7f2ce2ffd640 10 monclient(hunting): my global_id is 4132 2024-03-25T18:16:18.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.907+0000 7f2ce2ffd640 1 -- 172.21.15.98:0/738370650 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2cd4004eb0 con 0x7f2cf4146450 2024-03-25T18:16:18.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.907+0000 7f2ce2ffd640 1 -- 172.21.15.98:0/738370650 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 738572883 0 0) 0x7f2cf41dab70 con 0x7f2cf40a9920 2024-03-25T18:16:18.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.907+0000 7f2ce2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:16:18.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.907+0000 7f2ce2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:18.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.907+0000 7f2ce2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:18.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.907+0000 7f2ce2ffd640 10 monclient(hunting): my global_id is 4133 2024-03-25T18:16:18.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.907+0000 7f2ce2ffd640 1 -- 172.21.15.98:0/738370650 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2cf41453d0 con 0x7f2cf40a9920 2024-03-25T18:16:18.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.907+0000 7f2ce2ffd640 1 -- 172.21.15.98:0/738370650 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2844317015 0 0) 0x7f2cf4144610 con 0x7f2cf40b45a0 2024-03-25T18:16:18.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.907+0000 7f2ce2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:16:18.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.907+0000 7f2ce2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:18.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.907+0000 7f2ce2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:18.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.907+0000 7f2ce2ffd640 10 monclient(hunting): my global_id is 4143 2024-03-25T18:16:18.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.907+0000 7f2ce2ffd640 1 -- 172.21.15.98:0/738370650 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2cf41dab70 con 0x7f2cf40b45a0 2024-03-25T18:16:18.911 INFO:tasks.ceph.mgr.y.smithi097.stderr:2024-03-25T18:16:18.908+0000 7f7cdf810040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-25T18:16:18.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.912+0000 7f2ce2ffd640 1 -- 172.21.15.98:0/738370650 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4155870157 0 0) 0x7f2cd4004eb0 con 0x7f2cf4146450 2024-03-25T18:16:18.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.912+0000 7f2ce2ffd640 1 -- 172.21.15.98:0/738370650 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2cf4144610 con 0x7f2cf4146450 2024-03-25T18:16:18.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.912+0000 7f2ce2ffd640 1 -- 172.21.15.98:0/738370650 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f2cec001bd0 con 0x7f2cf4146450 2024-03-25T18:16:18.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.912+0000 7f2ce2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:18.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.912+0000 7f2ce2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:16:18.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.912+0000 7f2ce2ffd640 10 monclient(hunting): dump: 2024-03-25T18:16:18.913 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-03-25T18:16:18.913 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:16:18.913 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:18.914 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:18.914 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:18.914 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-03-25T18:16:18.914 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:16:18.914 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:16:18.914 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:16:18.914 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-25T18:16:18.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.912+0000 7f2ce2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:16:18.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.912+0000 7f2ce2ffd640 1 -- 172.21.15.98:0/738370650 <== mon.2 v1:172.21.15.97:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1860172085 0 0) 0x7f2cf41453d0 con 0x7f2cf40a9920 2024-03-25T18:16:18.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.912+0000 7f2ce2ffd640 1 -- 172.21.15.98:0/738370650 --> v1:172.21.15.97:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2cd4004eb0 con 0x7f2cf40a9920 2024-03-25T18:16:18.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.912+0000 7f2ce2ffd640 1 -- 172.21.15.98:0/738370650 <== mon.2 v1:172.21.15.97:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f2ce40018f0 con 0x7f2cf40a9920 2024-03-25T18:16:18.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.912+0000 7f2ce2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:18.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.912+0000 7f2ce2ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:16:18.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.912+0000 7f2ce2ffd640 10 monclient(hunting): dump: 2024-03-25T18:16:18.914 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-03-25T18:16:18.914 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:16:18.915 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:18.915 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:18.915 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:18.915 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-03-25T18:16:18.915 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:16:18.915 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:16:18.915 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:16:18.915 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-25T18:16:18.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.914+0000 7f2ce2ffd640 1 -- 172.21.15.98:0/738370650 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2528004921 0 0) 0x7f2cf41dab70 con 0x7f2cf40b45a0 2024-03-25T18:16:18.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.914+0000 7f2ce2ffd640 1 -- 172.21.15.98:0/738370650 --> v1:172.21.15.97:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2cf41453d0 con 0x7f2cf40b45a0 2024-03-25T18:16:18.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.914+0000 7f2ce2ffd640 1 -- 172.21.15.98:0/738370650 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f2ce8001430 con 0x7f2cf40b45a0 2024-03-25T18:16:18.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.914+0000 7f2ce2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:18.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.914+0000 7f2ce2ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:16:18.917 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.914+0000 7f2ce2ffd640 10 monclient(hunting): dump: 2024-03-25T18:16:18.917 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-03-25T18:16:18.917 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:16:18.917 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:18.917 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:18.917 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:18.917 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-03-25T18:16:18.917 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:16:18.917 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:16:18.917 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:16:18.918 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-25T18:16:18.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.915+0000 7f2ce2ffd640 1 -- 172.21.15.98:0/738370650 <== mon.1 v1:172.21.15.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1586121535 0 0) 0x7f2cf4144610 con 0x7f2cf4146450 2024-03-25T18:16:18.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.915+0000 7f2ce2ffd640 1 -- 172.21.15.98:0/738370650 >> v1:172.21.15.97:6790/0 conn(0x7f2cf40a9920 legacy=0x7f2cf4142830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:18.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.915+0000 7f2ce2ffd640 1 -- 172.21.15.98:0/738370650 >> v1:172.21.15.97:6789/0 conn(0x7f2cf40b45a0 legacy=0x7f2cf41db0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:18.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.915+0000 7f2ce2ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:16:18.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.915+0000 7f2ce2ffd640 1 monclient: found mon.b 2024-03-25T18:16:18.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.915+0000 7f2ce2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:18.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.915+0000 7f2ce2ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:16:18.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.915+0000 7f2ce2ffd640 1 -- 172.21.15.98:0/738370650 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cf4142f40 con 0x7f2cf4146450 2024-03-25T18:16:18.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.915+0000 7f2ce2ffd640 10 monclient: _finish_auth 0 2024-03-25T18:16:18.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.915+0000 7f2ce2ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:16:18.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.915+0000 7f2ce2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:18.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.915+0000 7f2cf8855640 5 monclient: authenticate success, global_id 4132 2024-03-25T18:16:18.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.915+0000 7f2cf8855640 10 monclient: _renew_subs 2024-03-25T18:16:18.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.915+0000 7f2cf8855640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:16:18.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.915+0000 7f2cf8855640 1 -- 172.21.15.98:0/738370650 --> v1:172.21.15.98:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cf4143190 con 0x7f2cf4146450 2024-03-25T18:16:18.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.915+0000 7f2cf8855640 10 client.4132.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:16:18.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.915+0000 7f2cf8855640 10 monclient: _renew_subs 2024-03-25T18:16:18.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.915+0000 7f2cf8855640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:16:18.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.915+0000 7f2cf8855640 1 -- 172.21.15.98:0/738370650 --> v1:172.21.15.98:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cf41db9c0 con 0x7f2cf4146450 2024-03-25T18:16:18.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.915+0000 7f2cf8855640 1 librados: init done 2024-03-25T18:16:18.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.916+0000 7f2cf8855640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:18.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.916+0000 7f2cf8855640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:18.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.916+0000 7f2cf8855640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:16:18.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:18.916+0000 7f2cf8855640 1 -- 172.21.15.98:0/738370650 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cf404f730 con 0x7f2cf4146450 2024-03-25T18:16:19.020 INFO:tasks.ceph.mgr.x.smithi098.stderr:2024-03-25T18:16:19.020+0000 7f16671db040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-25T18:16:19.064 INFO:tasks.ceph.mgr.x.smithi098.stderr:2024-03-25T18:16:19.063+0000 7f16671db040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-25T18:16:19.064 INFO:tasks.ceph.mgr.y.smithi097.stderr:2024-03-25T18:16:19.063+0000 7f7cdf810040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-25T18:16:19.106 INFO:tasks.ceph.mgr.y.smithi097.stderr:2024-03-25T18:16:19.105+0000 7f7cdf810040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-25T18:16:19.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.108+0000 7f2ce2ffd640 1 -- 172.21.15.98:0/738370650 <== mon.1 v1:172.21.15.98:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2cec0020e0 con 0x7f2cf4146450 2024-03-25T18:16:19.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.108+0000 7f2ce2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:19.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.108+0000 7f2ce2ffd640 1 -- 172.21.15.98:0/738370650 <== mon.1 v1:172.21.15.98:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f2cec003b10 con 0x7f2cf4146450 2024-03-25T18:16:19.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.108+0000 7f2ce2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:19.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.108+0000 7f2ce2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:16:19.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.108+0000 7f2ce2ffd640 10 monclient: dump: 2024-03-25T18:16:19.110 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-03-25T18:16:19.110 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:16:19.110 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:19.110 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:19.110 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:19.110 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-03-25T18:16:19.110 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:16:19.110 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:16:19.110 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:16:19.110 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-25T18:16:19.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.108+0000 7f2cf0d99640 4 set_mon_vals no callback set 2024-03-25T18:16:19.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.108+0000 7f2ce2ffd640 1 -- 172.21.15.98:0/738370650 <== mon.1 v1:172.21.15.98:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f2cec0040c0 con 0x7f2cf4146450 2024-03-25T18:16:19.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.108+0000 7f2ce2ffd640 1 -- 172.21.15.98:0/738370650 <== mon.1 v1:172.21.15.98:6789/0 8 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (unknown 522887750 0 0) 0x7f2cec0057d0 con 0x7f2cf4146450 2024-03-25T18:16:19.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.108+0000 7f2ce2ffd640 10 client.4132.objecter ms_dispatch 0x7f2cf4000b90 osd_map(6..6 src has 1..6) v4 2024-03-25T18:16:19.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.108+0000 7f2ce2ffd640 3 client.4132.objecter handle_osd_map got epochs [6,6] > 0 2024-03-25T18:16:19.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.108+0000 7f2ce2ffd640 3 client.4132.objecter handle_osd_map decoding full epoch 6 2024-03-25T18:16:19.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.108+0000 7f2ce2ffd640 20 client.4132.objecter dump_active .. 0 homeless 2024-03-25T18:16:19.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.108+0000 7f2ce2ffd640 1 -- 172.21.15.98:0/738370650 <== mon.1 v1:172.21.15.98:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f2cec005bd0 con 0x7f2cf4146450 2024-03-25T18:16:19.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.108+0000 7f2ce2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:19.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.108+0000 7f2ce2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:16:19.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.179+0000 7f2cf8855640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "f4f1df08-663f-4bda-93f8-3878ab7c24b3", "id": 4}] 2024-03-25T18:16:19.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.179+0000 7f2cf8855640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "f4f1df08-663f-4bda-93f8-3878ab7c24b3", "id": 4}] 2024-03-25T18:16:19.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.179+0000 7f2cf8855640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:16:19.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.179+0000 7f2cf8855640 1 -- 172.21.15.98:0/738370650 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "osd new", "uuid": "f4f1df08-663f-4bda-93f8-3878ab7c24b3", "id": 4} v 0) v1 -- 0x7f2cf413ea50 con 0x7f2cf4146450 2024-03-25T18:16:19.222 INFO:tasks.ceph.mgr.x.smithi098.stderr:2024-03-25T18:16:19.221+0000 7f16671db040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-25T18:16:19.266 INFO:tasks.ceph.mgr.y.smithi097.stderr:2024-03-25T18:16:19.265+0000 7f7cdf810040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-25T18:16:19.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.279+0000 7f2ce2ffd640 1 -- 172.21.15.98:0/738370650 <== mon.1 v1:172.21.15.98:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "f4f1df08-663f-4bda-93f8-3878ab7c24b3", "id": 4}]=0 v7) v1 ==== 112+0+2 (unknown 202002835 0 1697012583) 0x7f2cec005390 con 0x7f2cf4146450 2024-03-25T18:16:19.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.279+0000 7f2ce2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "f4f1df08-663f-4bda-93f8-3878ab7c24b3", "id": 4}] 2024-03-25T18:16:19.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.279+0000 7f2ce2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:16:19.280 INFO:teuthology.orchestra.run.smithi098.stdout:4 2024-03-25T18:16:19.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.280+0000 7f2cf8855640 10 librados: watch_flush enter 2024-03-25T18:16:19.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.280+0000 7f2cf8855640 10 librados: watch_flush exit 2024-03-25T18:16:19.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.280+0000 7f2cf8855640 20 client.4132.objecter shutdown clearing up homeless session... 2024-03-25T18:16:19.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.280+0000 7f2cf8855640 10 client.4132.objecter successfully canceled tick 2024-03-25T18:16:19.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.280+0000 7f2cf8855640 10 monclient: shutdown 2024-03-25T18:16:19.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.280+0000 7f2cf8855640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:19.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.280+0000 7f2cf8855640 1 -- 172.21.15.98:0/738370650 >> v1:172.21.15.98:6789/0 conn(0x7f2cf4146450 legacy=0x7f2cf41e2f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:19.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.280+0000 7f2cf8855640 1 -- 172.21.15.98:0/738370650 shutdown_connections 2024-03-25T18:16:19.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.280+0000 7f2cf8855640 1 -- 172.21.15.98:0/738370650 >> 172.21.15.98:0/738370650 conn(0x7f2cf40b7f40 msgr2=0x7f2cf4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:19.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.280+0000 7f2cf8855640 1 -- 172.21.15.98:0/738370650 shutdown_connections 2024-03-25T18:16:19.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.280+0000 7f2cf8855640 1 -- 172.21.15.98:0/738370650 wait complete. 2024-03-25T18:16:19.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.280+0000 7f2cf8855640 1 librados: shutdown 2024-03-25T18:16:19.292 DEBUG:teuthology.orchestra.run.smithi098:> sudo ceph --cluster ceph osd new c4b18d70-a3ce-4b15-9b70-eae5525bd761 5 2024-03-25T18:16:19.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.390+0000 7f1573655640 10 monclient: get_monmap_and_config 2024-03-25T18:16:19.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.390+0000 7f1573655640 10 monclient: build_initial_monmap 2024-03-25T18:16:19.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.390+0000 7f1573655640 10 monclient: monmap: 2024-03-25T18:16:19.391 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-03-25T18:16:19.391 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:16:19.391 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-03-25T18:16:19.391714+0000 2024-03-25T18:16:19.391 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-03-25T18:16:19.391714+0000 2024-03-25T18:16:19.391 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:19.392 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-03-25T18:16:19.392 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:16:19.392 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T18:16:19.392 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.97:6790/0 mon.noname-b 2024-03-25T18:16:19.392 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-25T18:16:19.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.391+0000 7f1573655640 1 Processor -- start 2024-03-25T18:16:19.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.391+0000 7f1573655640 1 -- start start 2024-03-25T18:16:19.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.391+0000 7f1573655640 10 monclient: init 2024-03-25T18:16:19.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.391+0000 7f1573655640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:19.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.392+0000 7f1573655640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:16:19.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.392+0000 7f1573655640 10 monclient(hunting): picked mon.noname-a con 0x7f156c0a9920 addr v1:172.21.15.97:6789/0 2024-03-25T18:16:19.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.392+0000 7f1573655640 10 monclient(hunting): picked mon.noname-c con 0x7f156c0b45a0 addr v1:172.21.15.98:6789/0 2024-03-25T18:16:19.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.392+0000 7f1573655640 10 monclient(hunting): picked mon.noname-b con 0x7f156c146450 addr v1:172.21.15.97:6790/0 2024-03-25T18:16:19.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.392+0000 7f1573655640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f156c06b100 con 0x7f156c0a9920 2024-03-25T18:16:19.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.392+0000 7f1573655640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f156c0b2940 con 0x7f156c0b45a0 2024-03-25T18:16:19.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.392+0000 7f1573655640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f156c13fec0 con 0x7f156c146450 2024-03-25T18:16:19.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.392+0000 7f1573655640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:19.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.392+0000 7f1573655640 10 monclient(hunting): authenticate will time out at 7579.530006s 2024-03-25T18:16:19.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.392+0000 7f1571bf2640 1 --1- >> v1:172.21.15.97:6790/0 conn(0x7f156c146450 0x7f156c14dc70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6790/0 says I am v1:172.21.15.98:45608/0 (socket says 172.21.15.98:45608) 2024-03-25T18:16:19.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.392+0000 7f1571bf2640 1 -- 172.21.15.98:0/1935561451 learned_addr learned my addr 172.21.15.98:0/1935561451 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-25T18:16:19.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.395+0000 7f1563fff640 1 -- 172.21.15.98:0/1935561451 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1419225603 0 0) 0x7f156c0b2940 con 0x7f156c0b45a0 2024-03-25T18:16:19.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.395+0000 7f1563fff640 10 monclient(hunting): _negotiate 2024-03-25T18:16:19.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.395+0000 7f1563fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:19.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.395+0000 7f1563fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:19.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.395+0000 7f1563fff640 10 monclient(hunting): my global_id is 4135 2024-03-25T18:16:19.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.395+0000 7f1563fff640 1 -- 172.21.15.98:0/1935561451 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1558003520 con 0x7f156c0b45a0 2024-03-25T18:16:19.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.395+0000 7f1563fff640 1 -- 172.21.15.98:0/1935561451 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1430897403 0 0) 0x7f156c06b100 con 0x7f156c0a9920 2024-03-25T18:16:19.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.395+0000 7f1563fff640 10 monclient(hunting): _negotiate 2024-03-25T18:16:19.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.395+0000 7f1563fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:19.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.395+0000 7f1563fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:19.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.395+0000 7f1563fff640 10 monclient(hunting): my global_id is 4146 2024-03-25T18:16:19.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.395+0000 7f1563fff640 1 -- 172.21.15.98:0/1935561451 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f156c0b2940 con 0x7f156c0a9920 2024-03-25T18:16:19.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.395+0000 7f1563fff640 1 -- 172.21.15.98:0/1935561451 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3280306534 0 0) 0x7f156c13fec0 con 0x7f156c146450 2024-03-25T18:16:19.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.395+0000 7f1563fff640 10 monclient(hunting): _negotiate 2024-03-25T18:16:19.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.395+0000 7f1563fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:19.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.395+0000 7f1563fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:19.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.395+0000 7f1563fff640 10 monclient(hunting): my global_id is 4136 2024-03-25T18:16:19.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.395+0000 7f1563fff640 1 -- 172.21.15.98:0/1935561451 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f156c06b100 con 0x7f156c146450 2024-03-25T18:16:19.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.398+0000 7f1563fff640 1 -- 172.21.15.98:0/1935561451 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1426230490 0 0) 0x7f156c0b2940 con 0x7f156c0a9920 2024-03-25T18:16:19.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.398+0000 7f1563fff640 1 -- 172.21.15.98:0/1935561451 >> v1:172.21.15.97:6790/0 conn(0x7f156c146450 legacy=0x7f156c14dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:19.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.398+0000 7f1563fff640 1 -- 172.21.15.98:0/1935561451 >> v1:172.21.15.98:6789/0 conn(0x7f156c0b45a0 legacy=0x7f156c145d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:19.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.398+0000 7f1563fff640 10 monclient: _finish_hunting 0 2024-03-25T18:16:19.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.398+0000 7f1563fff640 1 monclient: found mon.noname-a 2024-03-25T18:16:19.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.398+0000 7f1563fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:19.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.398+0000 7f1563fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.97:6789/0 2024-03-25T18:16:19.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.398+0000 7f1563fff640 1 -- 172.21.15.98:0/1935561451 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f156c140050 con 0x7f156c0a9920 2024-03-25T18:16:19.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.398+0000 7f1563fff640 10 monclient: _finish_auth 0 2024-03-25T18:16:19.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.398+0000 7f1563fff640 10 monclient: _check_auth_tickets 2024-03-25T18:16:19.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.398+0000 7f1563fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:19.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.398+0000 7f1573655640 5 monclient: authenticate success, global_id 4146 2024-03-25T18:16:19.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.398+0000 7f1573655640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:19.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.398+0000 7f1563fff640 1 -- 172.21.15.98:0/1935561451 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f1568001490 con 0x7f156c0a9920 2024-03-25T18:16:19.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.398+0000 7f1563fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:19.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.398+0000 7f1563fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:16:19.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.398+0000 7f1563fff640 10 monclient: dump: 2024-03-25T18:16:19.400 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-03-25T18:16:19.400 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:16:19.400 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:19.401 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:19.401 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:19.401 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-03-25T18:16:19.401 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:16:19.401 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:16:19.401 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:16:19.401 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-25T18:16:19.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.398+0000 7f1573655640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:19.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.400+0000 7f1563fff640 1 -- 172.21.15.98:0/1935561451 <== mon.0 v1:172.21.15.97:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f15680016e0 con 0x7f156c0a9920 2024-03-25T18:16:19.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.400+0000 7f1563fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:19.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.400+0000 7f1573655640 10 monclient: get_monmap_and_config success 2024-03-25T18:16:19.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.400+0000 7f1573655640 4 set_mon_vals no callback set 2024-03-25T18:16:19.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.400+0000 7f1563fff640 1 -- 172.21.15.98:0/1935561451 <== mon.0 v1:172.21.15.97:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f1568001a70 con 0x7f156c0a9920 2024-03-25T18:16:19.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.400+0000 7f1563fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:19.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.400+0000 7f1563fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:16:19.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.400+0000 7f1563fff640 10 monclient: dump: 2024-03-25T18:16:19.404 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-03-25T18:16:19.404 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:16:19.404 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:19.404 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:19.404 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:19.404 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-03-25T18:16:19.404 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:16:19.404 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:16:19.405 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:16:19.405 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-25T18:16:19.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.400+0000 7f1573655640 10 monclient: shutdown 2024-03-25T18:16:19.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.400+0000 7f1573655640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:19.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.400+0000 7f1573655640 1 -- 172.21.15.98:0/1935561451 >> v1:172.21.15.97:6789/0 conn(0x7f156c0a9920 legacy=0x7f156c13f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:19.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.400+0000 7f1573655640 1 -- 172.21.15.98:0/1935561451 shutdown_connections 2024-03-25T18:16:19.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.400+0000 7f1573655640 1 -- 172.21.15.98:0/1935561451 >> 172.21.15.98:0/1935561451 conn(0x7f156c0b7f40 msgr2=0x7f156c0b8350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:19.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.400+0000 7f1573655640 1 -- 172.21.15.98:0/1935561451 shutdown_connections 2024-03-25T18:16:19.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.400+0000 7f1573655640 1 -- 172.21.15.98:0/1935561451 wait complete. 2024-03-25T18:16:19.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.401+0000 7f1573655640 10 monclient: build_initial_monmap 2024-03-25T18:16:19.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.401+0000 7f1573655640 10 monclient: monmap: 2024-03-25T18:16:19.405 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-03-25T18:16:19.405 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:16:19.405 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-03-25T18:16:19.405 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-03-25T18:16:19.405 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:19.406 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-03-25T18:16:19.406 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:16:19.406 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T18:16:19.406 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.97:6790/0 mon.noname-c 2024-03-25T18:16:19.406 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-25T18:16:19.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.401+0000 7f1573655640 1 librados: starting msgr at 2024-03-25T18:16:19.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.401+0000 7f1573655640 1 librados: starting objecter 2024-03-25T18:16:19.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.401+0000 7f1573655640 1 Processor -- start 2024-03-25T18:16:19.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.401+0000 7f1573655640 1 -- start start 2024-03-25T18:16:19.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.401+0000 7f1573655640 1 librados: setting wanted keys 2024-03-25T18:16:19.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.401+0000 7f1573655640 1 librados: calling monclient init 2024-03-25T18:16:19.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.401+0000 7f1573655640 10 monclient: init 2024-03-25T18:16:19.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.402+0000 7f1573655640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:19.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.402+0000 7f1573655640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:16:19.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.402+0000 7f1573655640 10 monclient(hunting): picked mon.noname-b con 0x7f156c0a9920 addr v1:172.21.15.98:6789/0 2024-03-25T18:16:19.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.402+0000 7f1573655640 10 monclient(hunting): picked mon.noname-c con 0x7f156c0b45a0 addr v1:172.21.15.97:6790/0 2024-03-25T18:16:19.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.402+0000 7f1573655640 10 monclient(hunting): picked mon.noname-a con 0x7f156c146450 addr v1:172.21.15.97:6789/0 2024-03-25T18:16:19.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.402+0000 7f1573655640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f156c1d5180 con 0x7f156c146450 2024-03-25T18:16:19.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.402+0000 7f1573655640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f156c1421b0 con 0x7f156c0a9920 2024-03-25T18:16:19.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.402+0000 7f1573655640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f156c142700 con 0x7f156c0b45a0 2024-03-25T18:16:19.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.402+0000 7f1573655640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:19.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.402+0000 7f1573655640 10 monclient(hunting): authenticate will time out at 7579.540297s 2024-03-25T18:16:19.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.402+0000 7f1570bf0640 1 --1- >> v1:172.21.15.97:6790/0 conn(0x7f156c0b45a0 0x7f156c1406b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.97:6790/0 says I am v1:172.21.15.98:45614/0 (socket says 172.21.15.98:45614) 2024-03-25T18:16:19.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.402+0000 7f1570bf0640 1 -- 172.21.15.98:0/1168929157 learned_addr learned my addr 172.21.15.98:0/1168929157 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-25T18:16:19.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.403+0000 7f1561ffb640 10 client.?.objecter ms_handle_connect 0x7f156c0b45a0 2024-03-25T18:16:19.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.403+0000 7f1561ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:19.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.403+0000 7f1561ffb640 10 client.?.objecter ms_handle_connect 0x7f156c0a9920 2024-03-25T18:16:19.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.403+0000 7f1561ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:19.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.403+0000 7f1561ffb640 10 client.?.objecter ms_handle_connect 0x7f156c146450 2024-03-25T18:16:19.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.403+0000 7f1561ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:19.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.404+0000 7f1561ffb640 1 -- 172.21.15.98:0/1168929157 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 816824786 0 0) 0x7f156c1421b0 con 0x7f156c0a9920 2024-03-25T18:16:19.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.404+0000 7f1561ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:16:19.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.404+0000 7f1561ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:19.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.404+0000 7f1561ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:19.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.404+0000 7f1561ffb640 10 monclient(hunting): my global_id is 4138 2024-03-25T18:16:19.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.405+0000 7f1561ffb640 1 -- 172.21.15.98:0/1168929157 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1550004eb0 con 0x7f156c0a9920 2024-03-25T18:16:19.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.405+0000 7f1561ffb640 1 -- 172.21.15.98:0/1168929157 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2353597015 0 0) 0x7f156c06b100 con 0x7f156c146450 2024-03-25T18:16:19.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.405+0000 7f1561ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:16:19.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.405+0000 7f1561ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:19.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.405+0000 7f1561ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:19.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.405+0000 7f1561ffb640 10 monclient(hunting): my global_id is 4149 2024-03-25T18:16:19.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.405+0000 7f1561ffb640 1 -- 172.21.15.98:0/1168929157 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f156c1421b0 con 0x7f156c146450 2024-03-25T18:16:19.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.406+0000 7f1561ffb640 1 -- 172.21.15.98:0/1168929157 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 220807924 0 0) 0x7f156c142700 con 0x7f156c0b45a0 2024-03-25T18:16:19.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.406+0000 7f1561ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:16:19.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.406+0000 7f1561ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:19.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.406+0000 7f1561ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:19.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.406+0000 7f1561ffb640 10 monclient(hunting): my global_id is 4139 2024-03-25T18:16:19.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.406+0000 7f1561ffb640 1 -- 172.21.15.98:0/1168929157 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f156c06b100 con 0x7f156c0b45a0 2024-03-25T18:16:19.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.409+0000 7f1561ffb640 1 -- 172.21.15.98:0/1168929157 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3107981074 0 0) 0x7f1550004eb0 con 0x7f156c0a9920 2024-03-25T18:16:19.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.409+0000 7f1561ffb640 1 -- 172.21.15.98:0/1168929157 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f156c142700 con 0x7f156c0a9920 2024-03-25T18:16:19.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.409+0000 7f1561ffb640 1 -- 172.21.15.98:0/1168929157 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f1568001490 con 0x7f156c0a9920 2024-03-25T18:16:19.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.409+0000 7f1561ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:19.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.409+0000 7f1561ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:16:19.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.409+0000 7f1561ffb640 10 monclient(hunting): dump: 2024-03-25T18:16:19.411 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-03-25T18:16:19.411 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:16:19.411 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:19.411 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:19.411 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:19.411 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-03-25T18:16:19.411 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:16:19.411 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:16:19.411 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:16:19.412 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-25T18:16:19.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.409+0000 7f1561ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:16:19.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.409+0000 7f1561ffb640 1 -- 172.21.15.98:0/1168929157 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 762509084 0 0) 0x7f156c1421b0 con 0x7f156c146450 2024-03-25T18:16:19.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.409+0000 7f1561ffb640 1 -- 172.21.15.98:0/1168929157 --> v1:172.21.15.97:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1550004eb0 con 0x7f156c146450 2024-03-25T18:16:19.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.409+0000 7f1561ffb640 1 -- 172.21.15.98:0/1168929157 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f15640019b0 con 0x7f156c146450 2024-03-25T18:16:19.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.409+0000 7f1561ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:19.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.409+0000 7f1561ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:16:19.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.409+0000 7f1561ffb640 10 monclient(hunting): dump: 2024-03-25T18:16:19.412 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-03-25T18:16:19.412 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:16:19.412 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:19.412 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:19.412 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:19.412 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-03-25T18:16:19.412 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:16:19.412 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:16:19.413 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:16:19.413 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-25T18:16:19.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.412+0000 7f1561ffb640 1 -- 172.21.15.98:0/1168929157 <== mon.2 v1:172.21.15.97:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3049918966 0 0) 0x7f156c06b100 con 0x7f156c0b45a0 2024-03-25T18:16:19.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.412+0000 7f1561ffb640 1 -- 172.21.15.98:0/1168929157 --> v1:172.21.15.97:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f156c1421b0 con 0x7f156c0b45a0 2024-03-25T18:16:19.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.412+0000 7f1561ffb640 1 -- 172.21.15.98:0/1168929157 <== mon.2 v1:172.21.15.97:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f155c001910 con 0x7f156c0b45a0 2024-03-25T18:16:19.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.412+0000 7f1561ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:19.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.412+0000 7f1561ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:16:19.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.412+0000 7f1561ffb640 10 monclient(hunting): dump: 2024-03-25T18:16:19.414 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-03-25T18:16:19.414 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:16:19.414 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:19.414 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:19.414 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:19.414 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-03-25T18:16:19.415 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:16:19.415 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:16:19.415 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:16:19.415 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-25T18:16:19.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.412+0000 7f1561ffb640 1 -- 172.21.15.98:0/1168929157 <== mon.1 v1:172.21.15.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2198772262 0 0) 0x7f156c142700 con 0x7f156c0a9920 2024-03-25T18:16:19.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.412+0000 7f1561ffb640 1 -- 172.21.15.98:0/1168929157 >> v1:172.21.15.97:6790/0 conn(0x7f156c0b45a0 legacy=0x7f156c1406b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:19.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.412+0000 7f1561ffb640 1 -- 172.21.15.98:0/1168929157 >> v1:172.21.15.97:6789/0 conn(0x7f156c146450 legacy=0x7f156c1e1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:19.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.412+0000 7f1561ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:16:19.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.412+0000 7f1561ffb640 1 monclient: found mon.b 2024-03-25T18:16:19.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.412+0000 7f1561ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:19.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.412+0000 7f1561ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:16:19.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.412+0000 7f1561ffb640 1 -- 172.21.15.98:0/1168929157 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f156c143330 con 0x7f156c0a9920 2024-03-25T18:16:19.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.412+0000 7f1561ffb640 10 monclient: _finish_auth 0 2024-03-25T18:16:19.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.412+0000 7f1561ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:16:19.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.412+0000 7f1561ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:19.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.412+0000 7f1573655640 5 monclient: authenticate success, global_id 4138 2024-03-25T18:16:19.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.412+0000 7f1573655640 10 monclient: _renew_subs 2024-03-25T18:16:19.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.412+0000 7f1573655640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:16:19.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.412+0000 7f1573655640 1 -- 172.21.15.98:0/1168929157 --> v1:172.21.15.98:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f156c140dc0 con 0x7f156c0a9920 2024-03-25T18:16:19.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.413+0000 7f1573655640 10 client.4138.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:16:19.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.413+0000 7f1573655640 10 monclient: _renew_subs 2024-03-25T18:16:19.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.413+0000 7f1573655640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:16:19.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.413+0000 7f1573655640 1 -- 172.21.15.98:0/1168929157 --> v1:172.21.15.98:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f156c13cae0 con 0x7f156c0a9920 2024-03-25T18:16:19.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.413+0000 7f1573655640 1 librados: init done 2024-03-25T18:16:19.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.414+0000 7f1573655640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:19.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.414+0000 7f1573655640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:19.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.414+0000 7f1573655640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:16:19.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.414+0000 7f1573655640 1 -- 172.21.15.98:0/1168929157 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f156c04f730 con 0x7f156c0a9920 2024-03-25T18:16:19.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.414+0000 7f1561ffb640 1 -- 172.21.15.98:0/1168929157 <== mon.1 v1:172.21.15.98:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f15680038a0 con 0x7f156c0a9920 2024-03-25T18:16:19.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.414+0000 7f1561ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:19.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.414+0000 7f15637fe640 4 set_mon_vals no callback set 2024-03-25T18:16:19.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.417+0000 7f1561ffb640 1 -- 172.21.15.98:0/1168929157 <== mon.1 v1:172.21.15.98:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f1568003bb0 con 0x7f156c0a9920 2024-03-25T18:16:19.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.417+0000 7f1561ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:19.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.417+0000 7f1561ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:16:19.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.417+0000 7f1561ffb640 10 monclient: dump: 2024-03-25T18:16:19.418 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-03-25T18:16:19.418 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:16:19.419 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:19.419 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:19.419 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:19.419 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-03-25T18:16:19.419 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:16:19.419 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:16:19.419 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:16:19.419 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-25T18:16:19.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.417+0000 7f1561ffb640 1 -- 172.21.15.98:0/1168929157 <== mon.1 v1:172.21.15.98:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f15680040a0 con 0x7f156c0a9920 2024-03-25T18:16:19.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.417+0000 7f1561ffb640 1 -- 172.21.15.98:0/1168929157 <== mon.1 v1:172.21.15.98:6789/0 8 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (unknown 2482930605 0 0) 0x7f15680057a0 con 0x7f156c0a9920 2024-03-25T18:16:19.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.417+0000 7f1561ffb640 10 client.4138.objecter ms_dispatch 0x7f156c000b90 osd_map(7..7 src has 1..7) v4 2024-03-25T18:16:19.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.417+0000 7f1561ffb640 3 client.4138.objecter handle_osd_map got epochs [7,7] > 0 2024-03-25T18:16:19.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.417+0000 7f1561ffb640 3 client.4138.objecter handle_osd_map decoding full epoch 7 2024-03-25T18:16:19.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.417+0000 7f1561ffb640 20 client.4138.objecter dump_active .. 0 homeless 2024-03-25T18:16:19.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.524+0000 7f1561ffb640 1 -- 172.21.15.98:0/1168929157 <== mon.1 v1:172.21.15.98:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f15680052d0 con 0x7f156c0a9920 2024-03-25T18:16:19.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.524+0000 7f1561ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:19.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.524+0000 7f1561ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:16:19.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.617+0000 7f1573655640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "c4b18d70-a3ce-4b15-9b70-eae5525bd761", "id": 5}] 2024-03-25T18:16:19.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.617+0000 7f1573655640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "c4b18d70-a3ce-4b15-9b70-eae5525bd761", "id": 5}] 2024-03-25T18:16:19.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.617+0000 7f1573655640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:16:19.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.617+0000 7f1573655640 1 -- 172.21.15.98:0/1168929157 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "osd new", "uuid": "c4b18d70-a3ce-4b15-9b70-eae5525bd761", "id": 5} v 0) v1 -- 0x7f156c0b65a0 con 0x7f156c0a9920 2024-03-25T18:16:19.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.708+0000 7f1561ffb640 1 -- 172.21.15.98:0/1168929157 <== mon.1 v1:172.21.15.98:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "c4b18d70-a3ce-4b15-9b70-eae5525bd761", "id": 5}]=0 v8) v1 ==== 112+0+2 (unknown 1562783643 0 1988412176) 0x7f15680055b0 con 0x7f156c0a9920 2024-03-25T18:16:19.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.708+0000 7f1561ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "c4b18d70-a3ce-4b15-9b70-eae5525bd761", "id": 5}] 2024-03-25T18:16:19.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.708+0000 7f1561ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:16:19.709 INFO:teuthology.orchestra.run.smithi098.stdout:5 2024-03-25T18:16:19.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.709+0000 7f1573655640 10 librados: watch_flush enter 2024-03-25T18:16:19.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.709+0000 7f1573655640 10 librados: watch_flush exit 2024-03-25T18:16:19.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.709+0000 7f1573655640 20 client.4138.objecter shutdown clearing up homeless session... 2024-03-25T18:16:19.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.709+0000 7f1573655640 10 client.4138.objecter successfully canceled tick 2024-03-25T18:16:19.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.709+0000 7f1573655640 10 monclient: shutdown 2024-03-25T18:16:19.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.709+0000 7f1573655640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:19.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.709+0000 7f1573655640 1 -- 172.21.15.98:0/1168929157 >> v1:172.21.15.98:6789/0 conn(0x7f156c0a9920 legacy=0x7f156c142c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:19.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.709+0000 7f1573655640 1 -- 172.21.15.98:0/1168929157 shutdown_connections 2024-03-25T18:16:19.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.709+0000 7f1573655640 1 -- 172.21.15.98:0/1168929157 >> 172.21.15.98:0/1168929157 conn(0x7f156c0b7f40 msgr2=0x7f156c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:19.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.709+0000 7f1573655640 1 -- 172.21.15.98:0/1168929157 shutdown_connections 2024-03-25T18:16:19.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.709+0000 7f1573655640 1 -- 172.21.15.98:0/1168929157 wait complete. 2024-03-25T18:16:19.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.709+0000 7f1573655640 1 librados: shutdown 2024-03-25T18:16:19.721 DEBUG:teuthology.orchestra.run.smithi098:> sudo ceph --cluster ceph osd new 8dd4033a-de0e-4f8a-96dc-59f3c1f825b0 6 2024-03-25T18:16:19.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.819+0000 7f9e1eb82640 10 monclient: get_monmap_and_config 2024-03-25T18:16:19.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.819+0000 7f9e1eb82640 10 monclient: build_initial_monmap 2024-03-25T18:16:19.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.819+0000 7f9e1eb82640 10 monclient: monmap: 2024-03-25T18:16:19.821 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-03-25T18:16:19.821 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:16:19.821 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-03-25T18:16:19.820964+0000 2024-03-25T18:16:19.821 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-03-25T18:16:19.820964+0000 2024-03-25T18:16:19.821 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:19.821 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-03-25T18:16:19.821 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:16:19.821 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T18:16:19.821 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.97:6790/0 mon.noname-b 2024-03-25T18:16:19.821 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-25T18:16:19.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.820+0000 7f9e1eb82640 1 Processor -- start 2024-03-25T18:16:19.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.820+0000 7f9e1eb82640 1 -- start start 2024-03-25T18:16:19.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.820+0000 7f9e1eb82640 10 monclient: init 2024-03-25T18:16:19.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.821+0000 7f9e1eb82640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:19.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.821+0000 7f9e1eb82640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:16:19.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.821+0000 7f9e1eb82640 10 monclient(hunting): picked mon.noname-a con 0x7f9e1806caa0 addr v1:172.21.15.97:6789/0 2024-03-25T18:16:19.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.821+0000 7f9e1eb82640 10 monclient(hunting): picked mon.noname-b con 0x7f9e180ae1f0 addr v1:172.21.15.97:6790/0 2024-03-25T18:16:19.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.821+0000 7f9e1eb82640 10 monclient(hunting): picked mon.noname-c con 0x7f9e1813fc00 addr v1:172.21.15.98:6789/0 2024-03-25T18:16:19.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.821+0000 7f9e1eb82640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e1806ee90 con 0x7f9e1806caa0 2024-03-25T18:16:19.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.821+0000 7f9e1eb82640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e180adda0 con 0x7f9e1813fc00 2024-03-25T18:16:19.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.821+0000 7f9e1eb82640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e180729f0 con 0x7f9e180ae1f0 2024-03-25T18:16:19.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.821+0000 7f9e1eb82640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:19.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.821+0000 7f9e1eb82640 10 monclient(hunting): authenticate will time out at 7579.959302s 2024-03-25T18:16:19.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.821+0000 7f9e17fff640 1 --1- >> v1:172.21.15.97:6790/0 conn(0x7f9e180ae1f0 0x7f9e18145930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6790/0 says I am v1:172.21.15.98:45624/0 (socket says 172.21.15.98:45624) 2024-03-25T18:16:19.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.821+0000 7f9e17fff640 1 -- 172.21.15.98:0/691368495 learned_addr learned my addr 172.21.15.98:0/691368495 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-25T18:16:19.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.821+0000 7f9e1d11f640 1 --1- 172.21.15.98:0/691368495 >> v1:172.21.15.98:6789/0 conn(0x7f9e1813fc00 0x7f9e18146040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:60728/0 (socket says 172.21.15.98:60728) 2024-03-25T18:16:19.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.824+0000 7f9e177fe640 1 -- 172.21.15.98:0/691368495 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3709628614 0 0) 0x7f9e180729f0 con 0x7f9e180ae1f0 2024-03-25T18:16:19.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.824+0000 7f9e177fe640 10 monclient(hunting): _negotiate 2024-03-25T18:16:19.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.824+0000 7f9e177fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:19.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.824+0000 7f9e177fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:19.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.824+0000 7f9e177fe640 10 monclient(hunting): my global_id is 4142 2024-03-25T18:16:19.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.824+0000 7f9e177fe640 1 -- 172.21.15.98:0/691368495 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e00003520 con 0x7f9e180ae1f0 2024-03-25T18:16:19.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.824+0000 7f9e177fe640 1 -- 172.21.15.98:0/691368495 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2658968473 0 0) 0x7f9e180adda0 con 0x7f9e1813fc00 2024-03-25T18:16:19.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.824+0000 7f9e177fe640 10 monclient(hunting): _negotiate 2024-03-25T18:16:19.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.824+0000 7f9e177fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:19.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.824+0000 7f9e177fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:19.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.824+0000 7f9e177fe640 10 monclient(hunting): my global_id is 4141 2024-03-25T18:16:19.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.824+0000 7f9e177fe640 1 -- 172.21.15.98:0/691368495 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e180729f0 con 0x7f9e1813fc00 2024-03-25T18:16:19.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.824+0000 7f9e177fe640 1 -- 172.21.15.98:0/691368495 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 366150983 0 0) 0x7f9e1806ee90 con 0x7f9e1806caa0 2024-03-25T18:16:19.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.824+0000 7f9e177fe640 10 monclient(hunting): _negotiate 2024-03-25T18:16:19.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.824+0000 7f9e177fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:19.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.824+0000 7f9e177fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:19.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.824+0000 7f9e177fe640 10 monclient(hunting): my global_id is 4152 2024-03-25T18:16:19.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.824+0000 7f9e177fe640 1 -- 172.21.15.98:0/691368495 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e180adda0 con 0x7f9e1806caa0 2024-03-25T18:16:19.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.827+0000 7f9e177fe640 1 -- 172.21.15.98:0/691368495 <== mon.2 v1:172.21.15.97:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2628945161 0 0) 0x7f9e00003520 con 0x7f9e180ae1f0 2024-03-25T18:16:19.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.827+0000 7f9e177fe640 1 -- 172.21.15.98:0/691368495 >> v1:172.21.15.98:6789/0 conn(0x7f9e1813fc00 legacy=0x7f9e18146040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:19.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.827+0000 7f9e177fe640 1 -- 172.21.15.98:0/691368495 >> v1:172.21.15.97:6789/0 conn(0x7f9e1806caa0 legacy=0x7f9e1813f4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:19.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.827+0000 7f9e177fe640 10 monclient: _finish_hunting 0 2024-03-25T18:16:19.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.827+0000 7f9e177fe640 1 monclient: found mon.noname-b 2024-03-25T18:16:19.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.827+0000 7f9e177fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:19.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.827+0000 7f9e177fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.97:6790/0 2024-03-25T18:16:19.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.827+0000 7f9e177fe640 1 -- 172.21.15.98:0/691368495 --> v1:172.21.15.97:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e18072b80 con 0x7f9e180ae1f0 2024-03-25T18:16:19.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.827+0000 7f9e177fe640 10 monclient: _finish_auth 0 2024-03-25T18:16:19.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.827+0000 7f9e177fe640 10 monclient: _check_auth_tickets 2024-03-25T18:16:19.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.827+0000 7f9e177fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:19.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.827+0000 7f9e1eb82640 5 monclient: authenticate success, global_id 4142 2024-03-25T18:16:19.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.827+0000 7f9e1eb82640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:19.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.827+0000 7f9e177fe640 1 -- 172.21.15.98:0/691368495 <== mon.2 v1:172.21.15.97:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f9e0c0014b0 con 0x7f9e180ae1f0 2024-03-25T18:16:19.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.827+0000 7f9e177fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:19.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.827+0000 7f9e177fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:16:19.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.827+0000 7f9e177fe640 10 monclient: dump: 2024-03-25T18:16:19.829 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-03-25T18:16:19.829 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:16:19.829 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:19.830 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:19.830 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:19.830 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-03-25T18:16:19.830 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:16:19.830 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:16:19.830 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:16:19.830 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-25T18:16:19.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.827+0000 7f9e1eb82640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:19.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.829+0000 7f9e177fe640 1 -- 172.21.15.98:0/691368495 <== mon.2 v1:172.21.15.97:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9e0c001690 con 0x7f9e180ae1f0 2024-03-25T18:16:19.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.829+0000 7f9e177fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:19.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.829+0000 7f9e177fe640 1 -- 172.21.15.98:0/691368495 <== mon.2 v1:172.21.15.97:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f9e0c001a20 con 0x7f9e180ae1f0 2024-03-25T18:16:19.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.829+0000 7f9e177fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:19.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.829+0000 7f9e177fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:16:19.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.829+0000 7f9e177fe640 10 monclient: dump: 2024-03-25T18:16:19.832 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-03-25T18:16:19.832 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:16:19.832 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:19.832 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:19.832 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:19.832 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-03-25T18:16:19.832 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:16:19.833 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:16:19.833 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:16:19.833 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-25T18:16:19.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.829+0000 7f9e1eb82640 10 monclient: get_monmap_and_config success 2024-03-25T18:16:19.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.829+0000 7f9e1eb82640 4 set_mon_vals no callback set 2024-03-25T18:16:19.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.830+0000 7f9e1eb82640 10 monclient: shutdown 2024-03-25T18:16:19.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.830+0000 7f9e1eb82640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:19.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.830+0000 7f9e1eb82640 1 -- 172.21.15.98:0/691368495 >> v1:172.21.15.97:6790/0 conn(0x7f9e180ae1f0 legacy=0x7f9e18145930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:19.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.830+0000 7f9e1eb82640 1 -- 172.21.15.98:0/691368495 shutdown_connections 2024-03-25T18:16:19.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.830+0000 7f9e1eb82640 1 -- 172.21.15.98:0/691368495 >> 172.21.15.98:0/691368495 conn(0x7f9e1806bcb0 msgr2=0x7f9e1806c0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:19.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.830+0000 7f9e1eb82640 1 -- 172.21.15.98:0/691368495 shutdown_connections 2024-03-25T18:16:19.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.830+0000 7f9e1eb82640 1 -- 172.21.15.98:0/691368495 wait complete. 2024-03-25T18:16:19.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.831+0000 7f9e1eb82640 10 monclient: build_initial_monmap 2024-03-25T18:16:19.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.831+0000 7f9e1eb82640 10 monclient: monmap: 2024-03-25T18:16:19.833 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-03-25T18:16:19.833 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:16:19.834 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-03-25T18:16:19.834 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-03-25T18:16:19.834 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:19.834 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-03-25T18:16:19.834 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:16:19.834 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T18:16:19.834 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.97:6790/0 mon.noname-c 2024-03-25T18:16:19.834 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-25T18:16:19.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.831+0000 7f9e1eb82640 1 librados: starting msgr at 2024-03-25T18:16:19.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.831+0000 7f9e1eb82640 1 librados: starting objecter 2024-03-25T18:16:19.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.831+0000 7f9e1eb82640 1 Processor -- start 2024-03-25T18:16:19.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.831+0000 7f9e1eb82640 1 -- start start 2024-03-25T18:16:19.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.831+0000 7f9e1eb82640 1 librados: setting wanted keys 2024-03-25T18:16:19.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.831+0000 7f9e1eb82640 1 librados: calling monclient init 2024-03-25T18:16:19.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.831+0000 7f9e1eb82640 10 monclient: init 2024-03-25T18:16:19.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.832+0000 7f9e1eb82640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:19.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.832+0000 7f9e1eb82640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:16:19.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.832+0000 7f9e1eb82640 10 monclient(hunting): picked mon.noname-b con 0x7f9e1806caa0 addr v1:172.21.15.98:6789/0 2024-03-25T18:16:19.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.832+0000 7f9e1eb82640 10 monclient(hunting): picked mon.noname-a con 0x7f9e180ae1f0 addr v1:172.21.15.97:6789/0 2024-03-25T18:16:19.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.832+0000 7f9e1eb82640 10 monclient(hunting): picked mon.noname-c con 0x7f9e1813fc00 addr v1:172.21.15.97:6790/0 2024-03-25T18:16:19.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.832+0000 7f9e1eb82640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e181d7560 con 0x7f9e180ae1f0 2024-03-25T18:16:19.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.832+0000 7f9e1eb82640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e181d5090 con 0x7f9e1806caa0 2024-03-25T18:16:19.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.832+0000 7f9e1eb82640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e18130bd0 con 0x7f9e1813fc00 2024-03-25T18:16:19.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.832+0000 7f9e1eb82640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:19.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.832+0000 7f9e1eb82640 10 monclient(hunting): authenticate will time out at 7579.970236s 2024-03-25T18:16:19.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.832+0000 7f9e1d11f640 1 --1- >> v1:172.21.15.97:6790/0 conn(0x7f9e1813fc00 0x7f9e1812f3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.97:6790/0 says I am v1:172.21.15.98:45634/0 (socket says 172.21.15.98:45634) 2024-03-25T18:16:19.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.832+0000 7f9e1d11f640 1 -- 172.21.15.98:0/1426712766 learned_addr learned my addr 172.21.15.98:0/1426712766 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-25T18:16:19.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.833+0000 7f9e157fa640 10 client.?.objecter ms_handle_connect 0x7f9e1813fc00 2024-03-25T18:16:19.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.833+0000 7f9e157fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:19.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.833+0000 7f9e157fa640 10 client.?.objecter ms_handle_connect 0x7f9e180ae1f0 2024-03-25T18:16:19.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.833+0000 7f9e157fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:19.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.833+0000 7f9e157fa640 10 client.?.objecter ms_handle_connect 0x7f9e1806caa0 2024-03-25T18:16:19.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.833+0000 7f9e157fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:19.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.834+0000 7f9e157fa640 1 -- 172.21.15.98:0/1426712766 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2681095375 0 0) 0x7f9e18130bd0 con 0x7f9e1813fc00 2024-03-25T18:16:19.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.834+0000 7f9e157fa640 10 monclient(hunting): _negotiate 2024-03-25T18:16:19.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.834+0000 7f9e157fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:19.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.834+0000 7f9e157fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:19.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.834+0000 7f9e157fa640 10 monclient(hunting): my global_id is 4145 2024-03-25T18:16:19.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.834+0000 7f9e157fa640 1 -- 172.21.15.98:0/1426712766 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e04004eb0 con 0x7f9e1813fc00 2024-03-25T18:16:19.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.835+0000 7f9e157fa640 1 -- 172.21.15.98:0/1426712766 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2902288313 0 0) 0x7f9e181d7560 con 0x7f9e180ae1f0 2024-03-25T18:16:19.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.835+0000 7f9e157fa640 10 monclient(hunting): _negotiate 2024-03-25T18:16:19.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.835+0000 7f9e157fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:19.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.835+0000 7f9e157fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:19.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.835+0000 7f9e157fa640 10 monclient(hunting): my global_id is 4155 2024-03-25T18:16:19.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.835+0000 7f9e157fa640 1 -- 172.21.15.98:0/1426712766 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e18130bd0 con 0x7f9e180ae1f0 2024-03-25T18:16:19.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.835+0000 7f9e157fa640 1 -- 172.21.15.98:0/1426712766 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4099423644 0 0) 0x7f9e181d5090 con 0x7f9e1806caa0 2024-03-25T18:16:19.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.835+0000 7f9e157fa640 10 monclient(hunting): _negotiate 2024-03-25T18:16:19.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.835+0000 7f9e157fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:19.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.835+0000 7f9e157fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:19.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.835+0000 7f9e157fa640 10 monclient(hunting): my global_id is 4144 2024-03-25T18:16:19.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.835+0000 7f9e157fa640 1 -- 172.21.15.98:0/1426712766 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e181d7560 con 0x7f9e1806caa0 2024-03-25T18:16:19.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.839+0000 7f9e157fa640 1 -- 172.21.15.98:0/1426712766 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3652691785 0 0) 0x7f9e18130bd0 con 0x7f9e180ae1f0 2024-03-25T18:16:19.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.839+0000 7f9e157fa640 1 -- 172.21.15.98:0/1426712766 --> v1:172.21.15.97:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9e181d5090 con 0x7f9e180ae1f0 2024-03-25T18:16:19.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.839+0000 7f9e157fa640 1 -- 172.21.15.98:0/1426712766 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f9e0c000fe0 con 0x7f9e180ae1f0 2024-03-25T18:16:19.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.839+0000 7f9e157fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:19.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.839+0000 7f9e157fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:16:19.841 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.839+0000 7f9e157fa640 10 monclient(hunting): dump: 2024-03-25T18:16:19.841 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-03-25T18:16:19.841 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:16:19.841 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:19.842 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:19.842 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:19.842 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-03-25T18:16:19.842 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:16:19.842 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:16:19.842 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:16:19.842 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-25T18:16:19.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.839+0000 7f9e157fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:16:19.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.839+0000 7f9e157fa640 1 -- 172.21.15.98:0/1426712766 <== mon.2 v1:172.21.15.97:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1407371930 0 0) 0x7f9e04004eb0 con 0x7f9e1813fc00 2024-03-25T18:16:19.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.839+0000 7f9e157fa640 1 -- 172.21.15.98:0/1426712766 --> v1:172.21.15.97:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9e18130bd0 con 0x7f9e1813fc00 2024-03-25T18:16:19.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.839+0000 7f9e157fa640 1 -- 172.21.15.98:0/1426712766 <== mon.2 v1:172.21.15.97:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f9e08001850 con 0x7f9e1813fc00 2024-03-25T18:16:19.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.839+0000 7f9e157fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:19.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.839+0000 7f9e157fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:16:19.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.839+0000 7f9e157fa640 10 monclient(hunting): dump: 2024-03-25T18:16:19.843 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-03-25T18:16:19.843 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:16:19.843 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:19.843 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:19.843 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:19.843 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-03-25T18:16:19.843 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:16:19.843 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:16:19.843 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:16:19.843 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-25T18:16:19.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.840+0000 7f9e157fa640 1 -- 172.21.15.98:0/1426712766 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 167492046 0 0) 0x7f9e181d7560 con 0x7f9e1806caa0 2024-03-25T18:16:19.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.840+0000 7f9e157fa640 1 -- 172.21.15.98:0/1426712766 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9e04004eb0 con 0x7f9e1806caa0 2024-03-25T18:16:19.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.840+0000 7f9e157fa640 1 -- 172.21.15.98:0/1426712766 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f9e10001bb0 con 0x7f9e1806caa0 2024-03-25T18:16:19.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.840+0000 7f9e157fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:19.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.840+0000 7f9e157fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:16:19.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.840+0000 7f9e157fa640 10 monclient(hunting): dump: 2024-03-25T18:16:19.844 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-03-25T18:16:19.844 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:16:19.844 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:19.844 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:19.844 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:19.844 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-03-25T18:16:19.844 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:16:19.844 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:16:19.844 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:16:19.844 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-25T18:16:19.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.842+0000 7f9e157fa640 1 -- 172.21.15.98:0/1426712766 <== mon.0 v1:172.21.15.97:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4262143229 0 0) 0x7f9e181d5090 con 0x7f9e180ae1f0 2024-03-25T18:16:19.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.842+0000 7f9e157fa640 1 -- 172.21.15.98:0/1426712766 >> v1:172.21.15.97:6790/0 conn(0x7f9e1813fc00 legacy=0x7f9e1812f3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:19.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.842+0000 7f9e157fa640 1 -- 172.21.15.98:0/1426712766 >> v1:172.21.15.98:6789/0 conn(0x7f9e1806caa0 legacy=0x7f9e181d7a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:19.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.842+0000 7f9e157fa640 10 monclient: _finish_hunting 0 2024-03-25T18:16:19.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.842+0000 7f9e157fa640 1 monclient: found mon.a 2024-03-25T18:16:19.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.842+0000 7f9e157fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:19.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.842+0000 7f9e157fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:16:19.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.842+0000 7f9e157fa640 1 -- 172.21.15.98:0/1426712766 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e181d8150 con 0x7f9e180ae1f0 2024-03-25T18:16:19.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.842+0000 7f9e157fa640 10 monclient: _finish_auth 0 2024-03-25T18:16:19.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.842+0000 7f9e157fa640 10 monclient: _check_auth_tickets 2024-03-25T18:16:19.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.842+0000 7f9e157fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:19.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.842+0000 7f9e1eb82640 5 monclient: authenticate success, global_id 4155 2024-03-25T18:16:19.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.842+0000 7f9e1eb82640 10 monclient: _renew_subs 2024-03-25T18:16:19.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.842+0000 7f9e1eb82640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:16:19.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.842+0000 7f9e1eb82640 1 -- 172.21.15.98:0/1426712766 --> v1:172.21.15.97:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e181e2c70 con 0x7f9e180ae1f0 2024-03-25T18:16:19.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.842+0000 7f9e1eb82640 10 client.4155.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:16:19.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.842+0000 7f9e1eb82640 10 monclient: _renew_subs 2024-03-25T18:16:19.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.842+0000 7f9e1eb82640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:16:19.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.842+0000 7f9e1eb82640 1 -- 172.21.15.98:0/1426712766 --> v1:172.21.15.97:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e181e2fe0 con 0x7f9e180ae1f0 2024-03-25T18:16:19.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.842+0000 7f9e1eb82640 1 librados: init done 2024-03-25T18:16:19.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.843+0000 7f9e1eb82640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:19.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.843+0000 7f9e1eb82640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:19.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.843+0000 7f9e1eb82640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:16:19.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.843+0000 7f9e1eb82640 1 -- 172.21.15.98:0/1426712766 --> v1:172.21.15.97:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e180682a0 con 0x7f9e180ae1f0 2024-03-25T18:16:19.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.937+0000 7f9e157fa640 1 -- 172.21.15.98:0/1426712766 <== mon.0 v1:172.21.15.97:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9e0c001140 con 0x7f9e180ae1f0 2024-03-25T18:16:19.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.937+0000 7f9e157fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:19.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.937+0000 7f9e157fa640 1 -- 172.21.15.98:0/1426712766 <== mon.0 v1:172.21.15.97:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f9e0c004c30 con 0x7f9e180ae1f0 2024-03-25T18:16:19.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.937+0000 7f9e157fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:19.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.937+0000 7f9e157fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:16:19.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.937+0000 7f9e157fa640 10 monclient: dump: 2024-03-25T18:16:19.938 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-03-25T18:16:19.938 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:16:19.939 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:19.939 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:19.939 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:19.939 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-03-25T18:16:19.939 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:16:19.939 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:16:19.939 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:16:19.939 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-25T18:16:19.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.937+0000 7f9e157fa640 1 -- 172.21.15.98:0/1426712766 <== mon.0 v1:172.21.15.97:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f9e0c005120 con 0x7f9e180ae1f0 2024-03-25T18:16:19.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.937+0000 7f9e16ffd640 4 set_mon_vals no callback set 2024-03-25T18:16:19.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.938+0000 7f9e157fa640 1 -- 172.21.15.98:0/1426712766 <== mon.0 v1:172.21.15.97:6789/0 8 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (unknown 2767614349 0 0) 0x7f9e0c0067d0 con 0x7f9e180ae1f0 2024-03-25T18:16:19.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.938+0000 7f9e157fa640 10 client.4155.objecter ms_dispatch 0x7f9e18000b90 osd_map(8..8 src has 1..8) v4 2024-03-25T18:16:19.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.938+0000 7f9e157fa640 3 client.4155.objecter handle_osd_map got epochs [8,8] > 0 2024-03-25T18:16:19.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.938+0000 7f9e157fa640 3 client.4155.objecter handle_osd_map decoding full epoch 8 2024-03-25T18:16:19.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.938+0000 7f9e157fa640 20 client.4155.objecter dump_active .. 0 homeless 2024-03-25T18:16:19.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.938+0000 7f9e157fa640 1 -- 172.21.15.98:0/1426712766 <== mon.0 v1:172.21.15.97:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f9e0c006ac0 con 0x7f9e180ae1f0 2024-03-25T18:16:19.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.938+0000 7f9e157fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:19.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:19.938+0000 7f9e157fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:16:20.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.011+0000 7f9e1eb82640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "8dd4033a-de0e-4f8a-96dc-59f3c1f825b0", "id": 6}] 2024-03-25T18:16:20.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.011+0000 7f9e1eb82640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "8dd4033a-de0e-4f8a-96dc-59f3c1f825b0", "id": 6}] 2024-03-25T18:16:20.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.011+0000 7f9e1eb82640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:16:20.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.011+0000 7f9e1eb82640 1 -- 172.21.15.98:0/1426712766 --> v1:172.21.15.97:6789/0 -- mon_command({"prefix": "osd new", "uuid": "8dd4033a-de0e-4f8a-96dc-59f3c1f825b0", "id": 6} v 0) v1 -- 0x7f9e1804f730 con 0x7f9e180ae1f0 2024-03-25T18:16:20.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.039+0000 7f9e157fa640 1 -- 172.21.15.98:0/1426712766 <== mon.0 v1:172.21.15.97:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "8dd4033a-de0e-4f8a-96dc-59f3c1f825b0", "id": 6}]=0 v9) v1 ==== 112+0+2 (unknown 3951107960 0 1113811849) 0x7f9e0c006310 con 0x7f9e180ae1f0 2024-03-25T18:16:20.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.039+0000 7f9e157fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "8dd4033a-de0e-4f8a-96dc-59f3c1f825b0", "id": 6}] 2024-03-25T18:16:20.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.039+0000 7f9e157fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:16:20.040 INFO:teuthology.orchestra.run.smithi098.stdout:6 2024-03-25T18:16:20.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.039+0000 7f9e1eb82640 10 librados: watch_flush enter 2024-03-25T18:16:20.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.040+0000 7f9e1eb82640 10 librados: watch_flush exit 2024-03-25T18:16:20.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.040+0000 7f9e1eb82640 20 client.4155.objecter shutdown clearing up homeless session... 2024-03-25T18:16:20.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.040+0000 7f9e1eb82640 10 client.4155.objecter successfully canceled tick 2024-03-25T18:16:20.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.040+0000 7f9e1eb82640 10 monclient: shutdown 2024-03-25T18:16:20.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.040+0000 7f9e1eb82640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:20.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.040+0000 7f9e1eb82640 1 -- 172.21.15.98:0/1426712766 >> v1:172.21.15.97:6789/0 conn(0x7f9e180ae1f0 legacy=0x7f9e1812eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:20.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.040+0000 7f9e1eb82640 1 -- 172.21.15.98:0/1426712766 shutdown_connections 2024-03-25T18:16:20.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.040+0000 7f9e1eb82640 1 -- 172.21.15.98:0/1426712766 >> 172.21.15.98:0/1426712766 conn(0x7f9e1806bcb0 msgr2=0x7f9e1813dd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:20.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.040+0000 7f9e1eb82640 1 -- 172.21.15.98:0/1426712766 shutdown_connections 2024-03-25T18:16:20.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.040+0000 7f9e1eb82640 1 -- 172.21.15.98:0/1426712766 wait complete. 2024-03-25T18:16:20.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.040+0000 7f9e1eb82640 1 librados: shutdown 2024-03-25T18:16:20.051 DEBUG:teuthology.orchestra.run.smithi098:> sudo ceph --cluster ceph osd new 0077722e-ffa6-4a3d-9891-929d9869b286 7 2024-03-25T18:16:20.085 INFO:tasks.ceph.mgr.x.smithi098.stderr:2024-03-25T18:16:20.084+0000 7f16671db040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-25T18:16:20.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.149+0000 7faefda86640 10 monclient: get_monmap_and_config 2024-03-25T18:16:20.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.149+0000 7faefda86640 10 monclient: build_initial_monmap 2024-03-25T18:16:20.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.149+0000 7faefda86640 10 monclient: monmap: 2024-03-25T18:16:20.150 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-03-25T18:16:20.150 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:16:20.150 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-03-25T18:16:20.150839+0000 2024-03-25T18:16:20.151 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-03-25T18:16:20.150839+0000 2024-03-25T18:16:20.151 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:20.151 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-03-25T18:16:20.151 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:16:20.151 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T18:16:20.151 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.97:6790/0 mon.noname-b 2024-03-25T18:16:20.151 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-25T18:16:20.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.150+0000 7faefda86640 1 Processor -- start 2024-03-25T18:16:20.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.150+0000 7faefda86640 1 -- start start 2024-03-25T18:16:20.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.150+0000 7faefda86640 10 monclient: init 2024-03-25T18:16:20.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.151+0000 7faefda86640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:20.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.151+0000 7faefda86640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:16:20.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.151+0000 7faefda86640 10 monclient(hunting): picked mon.noname-b con 0x7faef80b4700 addr v1:172.21.15.97:6790/0 2024-03-25T18:16:20.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.151+0000 7faefda86640 10 monclient(hunting): picked mon.noname-a con 0x7faef8142a20 addr v1:172.21.15.97:6789/0 2024-03-25T18:16:20.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.151+0000 7faefda86640 10 monclient(hunting): picked mon.noname-c con 0x7faef80a96b0 addr v1:172.21.15.98:6789/0 2024-03-25T18:16:20.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.151+0000 7faefda86640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faef80a9d60 con 0x7faef8142a20 2024-03-25T18:16:20.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.151+0000 7faefda86640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faef813ff10 con 0x7faef80a96b0 2024-03-25T18:16:20.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.151+0000 7faefda86640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faef80aa1f0 con 0x7faef80b4700 2024-03-25T18:16:20.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.151+0000 7faefda86640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:20.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.151+0000 7faefda86640 10 monclient(hunting): authenticate will time out at 7580.289234s 2024-03-25T18:16:20.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.151+0000 7faef77fe640 1 --1- >> v1:172.21.15.97:6790/0 conn(0x7faef80b4700 0x7faef8142310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6790/0 says I am v1:172.21.15.98:45638/0 (socket says 172.21.15.98:45638) 2024-03-25T18:16:20.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.151+0000 7faef77fe640 1 -- 172.21.15.98:0/315663656 learned_addr learned my addr 172.21.15.98:0/315663656 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-25T18:16:20.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.154+0000 7faef6ffd640 1 -- 172.21.15.98:0/315663656 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3733749666 0 0) 0x7faef80aa1f0 con 0x7faef80b4700 2024-03-25T18:16:20.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.154+0000 7faef6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:16:20.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.154+0000 7faef6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:20.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.154+0000 7faef6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:20.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.154+0000 7faef6ffd640 10 monclient(hunting): my global_id is 4148 2024-03-25T18:16:20.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.154+0000 7faef6ffd640 1 -- 172.21.15.98:0/315663656 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faee4003520 con 0x7faef80b4700 2024-03-25T18:16:20.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.154+0000 7faef6ffd640 1 -- 172.21.15.98:0/315663656 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1539242372 0 0) 0x7faef813ff10 con 0x7faef80a96b0 2024-03-25T18:16:20.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.154+0000 7faef6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:16:20.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.154+0000 7faef6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:20.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.154+0000 7faef6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:20.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.154+0000 7faef6ffd640 10 monclient(hunting): my global_id is 4147 2024-03-25T18:16:20.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.154+0000 7faef6ffd640 1 -- 172.21.15.98:0/315663656 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faef80aa1f0 con 0x7faef80a96b0 2024-03-25T18:16:20.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.154+0000 7faef6ffd640 1 -- 172.21.15.98:0/315663656 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2473390333 0 0) 0x7faef80a9d60 con 0x7faef8142a20 2024-03-25T18:16:20.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.154+0000 7faef6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:16:20.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.154+0000 7faef6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:20.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.154+0000 7faef6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:20.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.154+0000 7faef6ffd640 10 monclient(hunting): my global_id is 4158 2024-03-25T18:16:20.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.155+0000 7faef6ffd640 1 -- 172.21.15.98:0/315663656 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faef813ff10 con 0x7faef8142a20 2024-03-25T18:16:20.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.157+0000 7faef6ffd640 1 -- 172.21.15.98:0/315663656 <== mon.2 v1:172.21.15.97:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1507536013 0 0) 0x7faee4003520 con 0x7faef80b4700 2024-03-25T18:16:20.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.157+0000 7faef6ffd640 1 -- 172.21.15.98:0/315663656 >> v1:172.21.15.98:6789/0 conn(0x7faef80a96b0 legacy=0x7faef813f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:20.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.157+0000 7faef6ffd640 1 -- 172.21.15.98:0/315663656 >> v1:172.21.15.97:6789/0 conn(0x7faef8142a20 legacy=0x7faef813f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:20.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.157+0000 7faef6ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:16:20.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.157+0000 7faef6ffd640 1 monclient: found mon.noname-b 2024-03-25T18:16:20.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.157+0000 7faef6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:20.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.157+0000 7faef6ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.97:6790/0 2024-03-25T18:16:20.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.157+0000 7faef6ffd640 1 -- 172.21.15.98:0/315663656 --> v1:172.21.15.97:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faef80b4ae0 con 0x7faef80b4700 2024-03-25T18:16:20.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.157+0000 7faef6ffd640 10 monclient: _finish_auth 0 2024-03-25T18:16:20.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.157+0000 7faef6ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:16:20.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.157+0000 7faef6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:20.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.157+0000 7faefda86640 5 monclient: authenticate success, global_id 4148 2024-03-25T18:16:20.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.157+0000 7faef6ffd640 1 -- 172.21.15.98:0/315663656 <== mon.2 v1:172.21.15.97:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7faee8001690 con 0x7faef80b4700 2024-03-25T18:16:20.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.157+0000 7faefda86640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:20.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.157+0000 7faef6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:20.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.157+0000 7faef6ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:16:20.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.157+0000 7faef6ffd640 10 monclient: dump: 2024-03-25T18:16:20.159 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-03-25T18:16:20.159 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:16:20.159 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:20.159 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:20.159 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:20.159 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-03-25T18:16:20.159 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:16:20.159 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:16:20.159 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:16:20.159 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-25T18:16:20.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.157+0000 7faefda86640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:20.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.159+0000 7faef6ffd640 1 -- 172.21.15.98:0/315663656 <== mon.2 v1:172.21.15.97:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faee80017f0 con 0x7faef80b4700 2024-03-25T18:16:20.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.159+0000 7faef6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:20.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.159+0000 7faef6ffd640 1 -- 172.21.15.98:0/315663656 <== mon.2 v1:172.21.15.97:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7faee8001c00 con 0x7faef80b4700 2024-03-25T18:16:20.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.159+0000 7faefda86640 10 monclient: get_monmap_and_config success 2024-03-25T18:16:20.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.159+0000 7faefda86640 4 set_mon_vals no callback set 2024-03-25T18:16:20.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.159+0000 7faef6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:20.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.159+0000 7faef6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:16:20.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.159+0000 7faef6ffd640 10 monclient: dump: 2024-03-25T18:16:20.161 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-03-25T18:16:20.161 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:16:20.161 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:20.161 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:20.161 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:20.161 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-03-25T18:16:20.161 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:16:20.161 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:16:20.161 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:16:20.161 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-25T18:16:20.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.159+0000 7faefda86640 10 monclient: shutdown 2024-03-25T18:16:20.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.159+0000 7faefda86640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:20.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.159+0000 7faefda86640 1 -- 172.21.15.98:0/315663656 >> v1:172.21.15.97:6790/0 conn(0x7faef80b4700 legacy=0x7faef8142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:20.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.159+0000 7faefda86640 1 -- 172.21.15.98:0/315663656 shutdown_connections 2024-03-25T18:16:20.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.159+0000 7faefda86640 1 -- 172.21.15.98:0/315663656 >> 172.21.15.98:0/315663656 conn(0x7faef80668e0 msgr2=0x7faef8066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:20.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.159+0000 7faefda86640 1 -- 172.21.15.98:0/315663656 shutdown_connections 2024-03-25T18:16:20.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.159+0000 7faefda86640 1 -- 172.21.15.98:0/315663656 wait complete. 2024-03-25T18:16:20.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.160+0000 7faefda86640 10 monclient: build_initial_monmap 2024-03-25T18:16:20.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.160+0000 7faefda86640 10 monclient: monmap: 2024-03-25T18:16:20.164 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-03-25T18:16:20.164 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:16:20.164 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-03-25T18:16:20.164 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-03-25T18:16:20.164 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:20.164 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-03-25T18:16:20.164 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:16:20.164 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T18:16:20.164 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.97:6790/0 mon.noname-c 2024-03-25T18:16:20.164 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-25T18:16:20.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.160+0000 7faefda86640 1 librados: starting msgr at 2024-03-25T18:16:20.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.160+0000 7faefda86640 1 librados: starting objecter 2024-03-25T18:16:20.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.160+0000 7faefda86640 1 Processor -- start 2024-03-25T18:16:20.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.161+0000 7faefda86640 1 -- start start 2024-03-25T18:16:20.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.161+0000 7faefda86640 1 librados: setting wanted keys 2024-03-25T18:16:20.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.161+0000 7faefda86640 1 librados: calling monclient init 2024-03-25T18:16:20.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.161+0000 7faefda86640 10 monclient: init 2024-03-25T18:16:20.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.161+0000 7faefda86640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:20.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.161+0000 7faefda86640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:16:20.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.161+0000 7faefda86640 10 monclient(hunting): picked mon.noname-c con 0x7faef80a96b0 addr v1:172.21.15.97:6790/0 2024-03-25T18:16:20.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.161+0000 7faefda86640 10 monclient(hunting): picked mon.noname-b con 0x7faef80b4700 addr v1:172.21.15.98:6789/0 2024-03-25T18:16:20.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.161+0000 7faefda86640 10 monclient(hunting): picked mon.noname-a con 0x7faef8142a20 addr v1:172.21.15.97:6789/0 2024-03-25T18:16:20.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.161+0000 7faefda86640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faef80aec00 con 0x7faef8142a20 2024-03-25T18:16:20.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.161+0000 7faefda86640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faef81d8300 con 0x7faef80b4700 2024-03-25T18:16:20.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.161+0000 7faefda86640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faef81d4fe0 con 0x7faef80a96b0 2024-03-25T18:16:20.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.161+0000 7faefda86640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:20.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.161+0000 7faefda86640 10 monclient(hunting): authenticate will time out at 7580.299552s 2024-03-25T18:16:20.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.162+0000 7faef77fe640 1 --1- >> v1:172.21.15.97:6790/0 conn(0x7faef80a96b0 0x7faef81d7bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.97:6790/0 says I am v1:172.21.15.98:45648/0 (socket says 172.21.15.98:45648) 2024-03-25T18:16:20.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.162+0000 7faef77fe640 1 -- 172.21.15.98:0/4120307350 learned_addr learned my addr 172.21.15.98:0/4120307350 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-25T18:16:20.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.162+0000 7faef4ff9640 10 client.?.objecter ms_handle_connect 0x7faef80a96b0 2024-03-25T18:16:20.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.162+0000 7faef4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:20.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.162+0000 7faef4ff9640 10 client.?.objecter ms_handle_connect 0x7faef80b4700 2024-03-25T18:16:20.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.162+0000 7faef4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:20.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.162+0000 7faef4ff9640 10 client.?.objecter ms_handle_connect 0x7faef8142a20 2024-03-25T18:16:20.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.162+0000 7faef4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:20.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.164+0000 7faef4ff9640 1 -- 172.21.15.98:0/4120307350 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3614023323 0 0) 0x7faef81d4fe0 con 0x7faef80a96b0 2024-03-25T18:16:20.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.164+0000 7faef4ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:16:20.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.164+0000 7faef4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:20.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.164+0000 7faef4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:20.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.164+0000 7faef4ff9640 10 monclient(hunting): my global_id is 4151 2024-03-25T18:16:20.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.164+0000 7faef4ff9640 1 -- 172.21.15.98:0/4120307350 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faed8004eb0 con 0x7faef80a96b0 2024-03-25T18:16:20.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.164+0000 7faef4ff9640 1 -- 172.21.15.98:0/4120307350 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1410512621 0 0) 0x7faef81d8300 con 0x7faef80b4700 2024-03-25T18:16:20.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.164+0000 7faef4ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:16:20.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.164+0000 7faef4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:20.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.164+0000 7faef4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:20.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.164+0000 7faef4ff9640 10 monclient(hunting): my global_id is 4150 2024-03-25T18:16:20.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.164+0000 7faef4ff9640 1 -- 172.21.15.98:0/4120307350 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faef81d4fe0 con 0x7faef80b4700 2024-03-25T18:16:20.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.165+0000 7faef4ff9640 1 -- 172.21.15.98:0/4120307350 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3981829611 0 0) 0x7faef80aec00 con 0x7faef8142a20 2024-03-25T18:16:20.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.165+0000 7faef4ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:16:20.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.165+0000 7faef4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:20.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.165+0000 7faef4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:20.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.165+0000 7faef4ff9640 10 monclient(hunting): my global_id is 4161 2024-03-25T18:16:20.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.165+0000 7faef4ff9640 1 -- 172.21.15.98:0/4120307350 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faef81d8300 con 0x7faef8142a20 2024-03-25T18:16:20.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.170+0000 7faef4ff9640 1 -- 172.21.15.98:0/4120307350 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2046889181 0 0) 0x7faef81d4fe0 con 0x7faef80b4700 2024-03-25T18:16:20.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.170+0000 7faef4ff9640 1 -- 172.21.15.98:0/4120307350 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faef80aec00 con 0x7faef80b4700 2024-03-25T18:16:20.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.170+0000 7faef4ff9640 1 -- 172.21.15.98:0/4120307350 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7faee00017d0 con 0x7faef80b4700 2024-03-25T18:16:20.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.170+0000 7faef4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:20.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.170+0000 7faef4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:16:20.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.170+0000 7faef4ff9640 10 monclient(hunting): dump: 2024-03-25T18:16:20.171 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-03-25T18:16:20.171 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:16:20.171 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:20.171 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:20.171 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:20.171 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-03-25T18:16:20.171 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:16:20.171 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:16:20.171 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:16:20.171 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-25T18:16:20.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.170+0000 7faef4ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:16:20.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.170+0000 7faef4ff9640 1 -- 172.21.15.98:0/4120307350 <== mon.2 v1:172.21.15.97:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3265655456 0 0) 0x7faed8004eb0 con 0x7faef80a96b0 2024-03-25T18:16:20.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.170+0000 7faef4ff9640 1 -- 172.21.15.98:0/4120307350 --> v1:172.21.15.97:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faef81d4fe0 con 0x7faef80a96b0 2024-03-25T18:16:20.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.170+0000 7faef4ff9640 1 -- 172.21.15.98:0/4120307350 <== mon.2 v1:172.21.15.97:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7faee8001bc0 con 0x7faef80a96b0 2024-03-25T18:16:20.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.170+0000 7faef4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:20.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.170+0000 7faef4ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:16:20.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.170+0000 7faef4ff9640 10 monclient(hunting): dump: 2024-03-25T18:16:20.173 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-03-25T18:16:20.173 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:16:20.173 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:20.173 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:20.173 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:20.173 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-03-25T18:16:20.173 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:16:20.173 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:16:20.173 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:16:20.173 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-25T18:16:20.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.171+0000 7faef4ff9640 1 -- 172.21.15.98:0/4120307350 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3475628831 0 0) 0x7faef81d8300 con 0x7faef8142a20 2024-03-25T18:16:20.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.171+0000 7faef4ff9640 1 -- 172.21.15.98:0/4120307350 --> v1:172.21.15.97:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faed8004eb0 con 0x7faef8142a20 2024-03-25T18:16:20.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.171+0000 7faef4ff9640 1 -- 172.21.15.98:0/4120307350 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7faef0001cb0 con 0x7faef8142a20 2024-03-25T18:16:20.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.171+0000 7faef4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:20.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.171+0000 7faef4ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:16:20.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.171+0000 7faef4ff9640 10 monclient(hunting): dump: 2024-03-25T18:16:20.174 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-03-25T18:16:20.174 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:16:20.174 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:20.174 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:20.174 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:20.174 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-03-25T18:16:20.174 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:16:20.174 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:16:20.174 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:16:20.174 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-25T18:16:20.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.174+0000 7faef4ff9640 1 -- 172.21.15.98:0/4120307350 <== mon.1 v1:172.21.15.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 793835354 0 0) 0x7faef80aec00 con 0x7faef80b4700 2024-03-25T18:16:20.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.174+0000 7faef4ff9640 1 -- 172.21.15.98:0/4120307350 >> v1:172.21.15.97:6790/0 conn(0x7faef80a96b0 legacy=0x7faef81d7bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:20.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.174+0000 7faef4ff9640 1 -- 172.21.15.98:0/4120307350 >> v1:172.21.15.97:6789/0 conn(0x7faef8142a20 legacy=0x7faef80ad590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:20.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.174+0000 7faef4ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:16:20.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.174+0000 7faef4ff9640 1 monclient: found mon.b 2024-03-25T18:16:20.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.174+0000 7faef4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:20.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.174+0000 7faef4ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:16:20.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.174+0000 7faef4ff9640 1 -- 172.21.15.98:0/4120307350 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faef81d73f0 con 0x7faef80b4700 2024-03-25T18:16:20.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.174+0000 7faef4ff9640 10 monclient: _finish_auth 0 2024-03-25T18:16:20.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.174+0000 7faef4ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:16:20.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.174+0000 7faef4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:20.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.174+0000 7faefda86640 5 monclient: authenticate success, global_id 4150 2024-03-25T18:16:20.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.174+0000 7faefda86640 10 monclient: _renew_subs 2024-03-25T18:16:20.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.174+0000 7faefda86640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:16:20.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.174+0000 7faefda86640 1 -- 172.21.15.98:0/4120307350 --> v1:172.21.15.98:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faef81d76d0 con 0x7faef80b4700 2024-03-25T18:16:20.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.174+0000 7faefda86640 10 client.4150.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:16:20.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.174+0000 7faefda86640 10 monclient: _renew_subs 2024-03-25T18:16:20.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.174+0000 7faefda86640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:16:20.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.174+0000 7faefda86640 1 -- 172.21.15.98:0/4120307350 --> v1:172.21.15.98:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7faef81e2f20 con 0x7faef80b4700 2024-03-25T18:16:20.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.174+0000 7faefda86640 1 librados: init done 2024-03-25T18:16:20.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.175+0000 7faefda86640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:20.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.175+0000 7faefda86640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:20.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.175+0000 7faefda86640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:16:20.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.175+0000 7faefda86640 1 -- 172.21.15.98:0/4120307350 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faef81d7910 con 0x7faef80b4700 2024-03-25T18:16:20.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.178+0000 7faef4ff9640 1 -- 172.21.15.98:0/4120307350 <== mon.1 v1:172.21.15.98:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faee0001ce0 con 0x7faef80b4700 2024-03-25T18:16:20.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.178+0000 7faef4ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:20.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.178+0000 7faef4ff9640 1 -- 172.21.15.98:0/4120307350 <== mon.1 v1:172.21.15.98:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7faee0003610 con 0x7faef80b4700 2024-03-25T18:16:20.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.178+0000 7faef4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:20.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.178+0000 7faef4ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:16:20.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.178+0000 7faef4ff9640 10 monclient: dump: 2024-03-25T18:16:20.180 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-03-25T18:16:20.180 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:16:20.180 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:20.180 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:20.180 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:20.180 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-03-25T18:16:20.180 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:16:20.180 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:16:20.180 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:16:20.180 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-25T18:16:20.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.178+0000 7faef67fc640 4 set_mon_vals no callback set 2024-03-25T18:16:20.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.178+0000 7faef4ff9640 1 -- 172.21.15.98:0/4120307350 <== mon.1 v1:172.21.15.98:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7faee0003b00 con 0x7faef80b4700 2024-03-25T18:16:20.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.178+0000 7faef4ff9640 1 -- 172.21.15.98:0/4120307350 <== mon.1 v1:172.21.15.98:6789/0 8 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (unknown 3568541659 0 0) 0x7faee0005370 con 0x7faef80b4700 2024-03-25T18:16:20.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.178+0000 7faef4ff9640 10 client.4150.objecter ms_dispatch 0x7faef8000b90 osd_map(9..9 src has 1..9) v4 2024-03-25T18:16:20.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.178+0000 7faef4ff9640 3 client.4150.objecter handle_osd_map got epochs [9,9] > 0 2024-03-25T18:16:20.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.178+0000 7faef4ff9640 3 client.4150.objecter handle_osd_map decoding full epoch 9 2024-03-25T18:16:20.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.178+0000 7faef4ff9640 20 client.4150.objecter dump_active .. 0 homeless 2024-03-25T18:16:20.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.273+0000 7faef4ff9640 1 -- 172.21.15.98:0/4120307350 <== mon.1 v1:172.21.15.98:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7faee0004e20 con 0x7faef80b4700 2024-03-25T18:16:20.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.273+0000 7faef4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:20.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.273+0000 7faef4ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:16:20.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.344+0000 7faefda86640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "0077722e-ffa6-4a3d-9891-929d9869b286", "id": 7}] 2024-03-25T18:16:20.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.344+0000 7faefda86640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "0077722e-ffa6-4a3d-9891-929d9869b286", "id": 7}] 2024-03-25T18:16:20.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.344+0000 7faefda86640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:16:20.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.344+0000 7faefda86640 1 -- 172.21.15.98:0/4120307350 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "osd new", "uuid": "0077722e-ffa6-4a3d-9891-929d9869b286", "id": 7} v 0) v1 -- 0x7faef804f730 con 0x7faef80b4700 2024-03-25T18:16:20.347 INFO:tasks.ceph.mgr.y.smithi097.stderr:2024-03-25T18:16:20.346+0000 7f7cdf810040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-25T18:16:20.370 INFO:tasks.ceph.mgr.x.smithi098.stderr:2024-03-25T18:16:20.370+0000 7f16671db040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-25T18:16:20.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.404+0000 7faef4ff9640 1 -- 172.21.15.98:0/4120307350 <== mon.1 v1:172.21.15.98:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "0077722e-ffa6-4a3d-9891-929d9869b286", "id": 7}]=0 v10) v1 ==== 112+0+2 (unknown 2031447581 0 1371665406) 0x7faee0005030 con 0x7faef80b4700 2024-03-25T18:16:20.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.404+0000 7faef4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "0077722e-ffa6-4a3d-9891-929d9869b286", "id": 7}] 2024-03-25T18:16:20.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.404+0000 7faef4ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:16:20.405 INFO:teuthology.orchestra.run.smithi098.stdout:7 2024-03-25T18:16:20.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.405+0000 7faefda86640 10 librados: watch_flush enter 2024-03-25T18:16:20.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.405+0000 7faefda86640 10 librados: watch_flush exit 2024-03-25T18:16:20.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.405+0000 7faefda86640 20 client.4150.objecter shutdown clearing up homeless session... 2024-03-25T18:16:20.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.405+0000 7faefda86640 10 client.4150.objecter successfully canceled tick 2024-03-25T18:16:20.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.405+0000 7faefda86640 10 monclient: shutdown 2024-03-25T18:16:20.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.405+0000 7faefda86640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:20.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.405+0000 7faefda86640 1 -- 172.21.15.98:0/4120307350 >> v1:172.21.15.98:6789/0 conn(0x7faef80b4700 legacy=0x7faef80ace80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:20.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.405+0000 7faefda86640 1 -- 172.21.15.98:0/4120307350 shutdown_connections 2024-03-25T18:16:20.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.405+0000 7faefda86640 1 -- 172.21.15.98:0/4120307350 >> 172.21.15.98:0/4120307350 conn(0x7faef80668e0 msgr2=0x7faef813d2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:20.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.405+0000 7faefda86640 1 -- 172.21.15.98:0/4120307350 shutdown_connections 2024-03-25T18:16:20.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.405+0000 7faefda86640 1 -- 172.21.15.98:0/4120307350 wait complete. 2024-03-25T18:16:20.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T18:16:20.405+0000 7faefda86640 1 librados: shutdown 2024-03-25T18:16:20.416 DEBUG:tasks.ceph_manager:running osd.0 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.0.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:16:20.416 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-25T18:16:20.416 DEBUG:teuthology.orchestra.run.smithi097:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 2024-03-25T18:16:20.419 INFO:tasks.ceph.osd.0:Started 2024-03-25T18:16:20.419 DEBUG:tasks.ceph_manager:running osd.1 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.1.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:16:20.419 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-25T18:16:20.419 DEBUG:teuthology.orchestra.run.smithi097:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 2024-03-25T18:16:20.421 INFO:tasks.ceph.osd.1:Started 2024-03-25T18:16:20.421 DEBUG:tasks.ceph_manager:running osd.2 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.2.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:16:20.421 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-25T18:16:20.421 DEBUG:teuthology.orchestra.run.smithi097:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 2024-03-25T18:16:20.423 INFO:tasks.ceph.osd.2:Started 2024-03-25T18:16:20.424 DEBUG:tasks.ceph_manager:running osd.3 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.3.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:16:20.424 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-25T18:16:20.424 DEBUG:teuthology.orchestra.run.smithi097:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 2024-03-25T18:16:20.425 INFO:tasks.ceph.mgr.x.smithi098.stderr:2024-03-25T18:16:20.423+0000 7f16671db040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-25T18:16:20.426 INFO:tasks.ceph.osd.3:Started 2024-03-25T18:16:20.426 DEBUG:tasks.ceph_manager:running osd.4 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.4.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:16:20.427 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-25T18:16:20.427 DEBUG:teuthology.orchestra.run.smithi098:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 2024-03-25T18:16:20.429 INFO:tasks.ceph.osd.4:Started 2024-03-25T18:16:20.429 DEBUG:tasks.ceph_manager:running osd.5 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.5.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:16:20.429 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-25T18:16:20.429 DEBUG:teuthology.orchestra.run.smithi098:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 2024-03-25T18:16:20.431 INFO:tasks.ceph.osd.5:Started 2024-03-25T18:16:20.431 DEBUG:tasks.ceph_manager:running osd.6 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.6.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:16:20.431 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-25T18:16:20.431 DEBUG:teuthology.orchestra.run.smithi098:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 2024-03-25T18:16:20.433 INFO:tasks.ceph.osd.6:Started 2024-03-25T18:16:20.433 DEBUG:tasks.ceph_manager:running osd.7 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.7.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:16:20.433 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-25T18:16:20.434 DEBUG:teuthology.orchestra.run.smithi098:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 2024-03-25T18:16:20.436 INFO:tasks.ceph.osd.7:Started 2024-03-25T18:16:20.436 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-25T18:16:20.577 INFO:tasks.ceph.mgr.x.smithi098.stderr:2024-03-25T18:16:20.577+0000 7f16671db040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-25T18:16:20.596 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.594+0000 7f2fa2b32640 10 monclient: get_monmap_and_config 2024-03-25T18:16:20.596 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.594+0000 7f2fa2b32640 10 monclient: build_initial_monmap 2024-03-25T18:16:20.596 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.594+0000 7f2fa2b32640 10 monclient: monmap: 2024-03-25T18:16:20.596 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 0 2024-03-25T18:16:20.596 INFO:teuthology.orchestra.run.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:16:20.596 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2024-03-25T18:16:20.596188+0000 2024-03-25T18:16:20.596 INFO:teuthology.orchestra.run.smithi097.stderr:created 2024-03-25T18:16:20.596188+0000 2024-03-25T18:16:20.596 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:20.596 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 3 2024-03-25T18:16:20.597 INFO:teuthology.orchestra.run.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:16:20.597 INFO:teuthology.orchestra.run.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T18:16:20.597 INFO:teuthology.orchestra.run.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-b 2024-03-25T18:16:20.597 INFO:teuthology.orchestra.run.smithi097.stderr: 2024-03-25T18:16:20.598 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.595+0000 7f2fa2b32640 1 Processor -- start 2024-03-25T18:16:20.599 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.596+0000 7f2fa2b32640 1 -- start start 2024-03-25T18:16:20.599 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.596+0000 7f2fa2b32640 10 monclient: init 2024-03-25T18:16:20.599 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.596+0000 7f2fa2b32640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:20.599 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.596+0000 7f2fa2b32640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:16:20.599 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.596+0000 7f2fa2b32640 10 monclient(hunting): picked mon.noname-a con 0x7f2f9c0a9920 addr v1:172.21.15.97:6789/0 2024-03-25T18:16:20.599 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.596+0000 7f2fa2b32640 10 monclient(hunting): picked mon.noname-c con 0x7f2f9c0b45a0 addr v1:172.21.15.98:6789/0 2024-03-25T18:16:20.599 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.596+0000 7f2fa2b32640 10 monclient(hunting): picked mon.noname-b con 0x7f2f9c146450 addr v1:172.21.15.97:6790/0 2024-03-25T18:16:20.599 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.596+0000 7f2fa2b32640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f9c06b100 con 0x7f2f9c0a9920 2024-03-25T18:16:20.599 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.597+0000 7f2fa2b32640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f9c0b2940 con 0x7f2f9c0b45a0 2024-03-25T18:16:20.599 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.597+0000 7f2fa2b32640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f9c13fec0 con 0x7f2f9c146450 2024-03-25T18:16:20.599 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.597+0000 7f2fa2b32640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:20.599 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.597+0000 7f2fa2b32640 10 monclient(hunting): authenticate will time out at 7592.872894s 2024-03-25T18:16:20.599 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.597+0000 7f2fa10cf640 1 --1- >> v1:172.21.15.97:6790/0 conn(0x7f2f9c146450 0x7f2f9c14dc70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6790/0 says I am v1:172.21.15.97:58996/0 (socket says 172.21.15.97:58996) 2024-03-25T18:16:20.600 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.597+0000 7f2fa10cf640 1 -- 172.21.15.97:0/2011706409 learned_addr learned my addr 172.21.15.97:0/2011706409 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:16:20.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.601+0000 7f2f9b7fe640 1 -- 172.21.15.97:0/2011706409 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 952236365 0 0) 0x7f2f9c0b2940 con 0x7f2f9c0b45a0 2024-03-25T18:16:20.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.601+0000 7f2f9b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:16:20.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.601+0000 7f2f9b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:20.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.601+0000 7f2f9b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:20.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.601+0000 7f2f9b7fe640 10 monclient(hunting): my global_id is 4153 2024-03-25T18:16:20.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.601+0000 7f2f9b7fe640 1 -- 172.21.15.97:0/2011706409 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f8c003520 con 0x7f2f9c0b45a0 2024-03-25T18:16:20.604 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.601+0000 7f2f9b7fe640 1 -- 172.21.15.97:0/2011706409 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2402933061 0 0) 0x7f2f9c13fec0 con 0x7f2f9c146450 2024-03-25T18:16:20.604 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.601+0000 7f2f9b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:16:20.604 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.601+0000 7f2f9b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:20.604 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.601+0000 7f2f9b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:20.604 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.601+0000 7f2f9b7fe640 10 monclient(hunting): my global_id is 4154 2024-03-25T18:16:20.604 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.601+0000 7f2f9b7fe640 1 -- 172.21.15.97:0/2011706409 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f9c0b2940 con 0x7f2f9c146450 2024-03-25T18:16:20.608 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.606+0000 7f2f9b7fe640 1 -- 172.21.15.97:0/2011706409 <== mon.2 v1:172.21.15.97:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2098069929 0 0) 0x7f2f9c0b2940 con 0x7f2f9c146450 2024-03-25T18:16:20.608 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.606+0000 7f2f9b7fe640 1 -- 172.21.15.97:0/2011706409 >> v1:172.21.15.98:6789/0 conn(0x7f2f9c0b45a0 legacy=0x7f2f9c145d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:20.608 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.606+0000 7f2f9b7fe640 1 -- 172.21.15.97:0/2011706409 >> v1:172.21.15.97:6789/0 conn(0x7f2f9c0a9920 legacy=0x7f2f9c13f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:16:20.609 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.606+0000 7f2f9b7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:16:20.609 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.606+0000 7f2f9b7fe640 1 monclient: found mon.noname-b 2024-03-25T18:16:20.609 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.606+0000 7f2f9b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:20.609 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.606+0000 7f2f9b7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.97:6790/0 2024-03-25T18:16:20.609 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.606+0000 7f2f9b7fe640 1 -- 172.21.15.97:0/2011706409 --> v1:172.21.15.97:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f9c140050 con 0x7f2f9c146450 2024-03-25T18:16:20.609 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.606+0000 7f2f9b7fe640 10 monclient: _finish_auth 0 2024-03-25T18:16:20.609 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.606+0000 7f2f9b7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:16:20.609 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.606+0000 7f2f9b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:20.609 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.606+0000 7f2fa2b32640 5 monclient: authenticate success, global_id 4154 2024-03-25T18:16:20.609 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.606+0000 7f2fa2b32640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:20.609 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.606+0000 7f2f9b7fe640 1 -- 172.21.15.97:0/2011706409 <== mon.2 v1:172.21.15.97:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f2f94001790 con 0x7f2f9c146450 2024-03-25T18:16:20.609 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.606+0000 7f2f9b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:20.609 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.606+0000 7f2f9b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:16:20.609 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.606+0000 7f2f9b7fe640 10 monclient: dump: 2024-03-25T18:16:20.610 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2024-03-25T18:16:20.610 INFO:teuthology.orchestra.run.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:16:20.610 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:20.610 INFO:teuthology.orchestra.run.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:20.610 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:20.610 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 3 2024-03-25T18:16:20.610 INFO:teuthology.orchestra.run.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:16:20.610 INFO:teuthology.orchestra.run.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:16:20.610 INFO:teuthology.orchestra.run.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:16:20.610 INFO:teuthology.orchestra.run.smithi097.stderr: 2024-03-25T18:16:20.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.606+0000 7f2fa2b32640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:20.612 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.608+0000 7f2f9b7fe640 1 -- 172.21.15.97:0/2011706409 <== mon.2 v1:172.21.15.97:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2f94001a20 con 0x7f2f9c146450 2024-03-25T18:16:20.612 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.608+0000 7f2f9b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:20.612 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.609+0000 7f2fa2b32640 10 monclient: get_monmap_and_config success 2024-03-25T18:16:20.612 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.609+0000 7f2fa2b32640 4 set_mon_vals no callback set 2024-03-25T18:16:20.612 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.609+0000 7f2f9b7fe640 1 -- 172.21.15.97:0/2011706409 <== mon.2 v1:172.21.15.97:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f2f94001db0 con 0x7f2f9c146450 2024-03-25T18:16:20.612 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.609+0000 7f2f9b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:20.612 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.609+0000 7f2f9b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:16:20.612 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.609+0000 7f2f9b7fe640 10 monclient: dump: 2024-03-25T18:16:20.613 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2024-03-25T18:16:20.613 INFO:teuthology.orchestra.run.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:16:20.613 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:20.613 INFO:teuthology.orchestra.run.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:20.613 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:20.613 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 3 2024-03-25T18:16:20.613 INFO:teuthology.orchestra.run.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:16:20.613 INFO:teuthology.orchestra.run.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:16:20.613 INFO:teuthology.orchestra.run.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:16:20.613 INFO:teuthology.orchestra.run.smithi097.stderr: 2024-03-25T18:16:20.613 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.609+0000 7f2fa2b32640 10 monclient: shutdown 2024-03-25T18:16:20.613 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.609+0000 7f2fa2b32640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:20.614 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.609+0000 7f2fa2b32640 1 -- 172.21.15.97:0/2011706409 >> v1:172.21.15.97:6790/0 conn(0x7f2f9c146450 legacy=0x7f2f9c14dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:20.614 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.610+0000 7f2fa2b32640 1 -- 172.21.15.97:0/2011706409 shutdown_connections 2024-03-25T18:16:20.614 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.610+0000 7f2fa2b32640 1 -- 172.21.15.97:0/2011706409 >> 172.21.15.97:0/2011706409 conn(0x7f2f9c0b7f40 msgr2=0x7f2f9c0b8350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:20.614 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.610+0000 7f2fa2b32640 1 -- 172.21.15.97:0/2011706409 shutdown_connections 2024-03-25T18:16:20.616 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.610+0000 7f2fa2b32640 1 -- 172.21.15.97:0/2011706409 wait complete. 2024-03-25T18:16:20.617 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.611+0000 7f2fa2b32640 10 monclient: build_initial_monmap 2024-03-25T18:16:20.617 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.611+0000 7f2fa2b32640 10 monclient: monmap: 2024-03-25T18:16:20.617 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 0 2024-03-25T18:16:20.617 INFO:teuthology.orchestra.run.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:16:20.617 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 0.000000 2024-03-25T18:16:20.617 INFO:teuthology.orchestra.run.smithi097.stderr:created 0.000000 2024-03-25T18:16:20.617 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:20.617 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 1 2024-03-25T18:16:20.617 INFO:teuthology.orchestra.run.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:16:20.617 INFO:teuthology.orchestra.run.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T18:16:20.617 INFO:teuthology.orchestra.run.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-c 2024-03-25T18:16:20.617 INFO:teuthology.orchestra.run.smithi097.stderr: 2024-03-25T18:16:20.618 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.612+0000 7f2fa2b32640 1 librados: starting msgr at 2024-03-25T18:16:20.618 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.612+0000 7f2fa2b32640 1 librados: starting objecter 2024-03-25T18:16:20.618 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.612+0000 7f2fa2b32640 1 Processor -- start 2024-03-25T18:16:20.618 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.612+0000 7f2fa2b32640 1 -- start start 2024-03-25T18:16:20.618 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.612+0000 7f2fa2b32640 1 librados: setting wanted keys 2024-03-25T18:16:20.618 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.612+0000 7f2fa2b32640 1 librados: calling monclient init 2024-03-25T18:16:20.618 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.612+0000 7f2fa2b32640 10 monclient: init 2024-03-25T18:16:20.618 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.613+0000 7f2fa2b32640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:20.618 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.613+0000 7f2fa2b32640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:16:20.618 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.613+0000 7f2fa2b32640 10 monclient(hunting): picked mon.noname-b con 0x7f2f9c0a9920 addr v1:172.21.15.98:6789/0 2024-03-25T18:16:20.618 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.613+0000 7f2fa2b32640 10 monclient(hunting): picked mon.noname-a con 0x7f2f9c0b45a0 addr v1:172.21.15.97:6789/0 2024-03-25T18:16:20.618 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.613+0000 7f2fa2b32640 10 monclient(hunting): picked mon.noname-c con 0x7f2f9c146450 addr v1:172.21.15.97:6790/0 2024-03-25T18:16:20.618 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.614+0000 7f2fa2b32640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f9c140750 con 0x7f2f9c0b45a0 2024-03-25T18:16:20.619 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.614+0000 7f2fa2b32640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f9c072000 con 0x7f2f9c0a9920 2024-03-25T18:16:20.619 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.614+0000 7f2fa2b32640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f9c13ba90 con 0x7f2f9c146450 2024-03-25T18:16:20.619 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.614+0000 7f2fa2b32640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:20.619 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.614+0000 7f2fa2b32640 10 monclient(hunting): authenticate will time out at 7592.890243s 2024-03-25T18:16:20.619 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.614+0000 7f2fa08ce640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f2f9c0a9920 0x7f2f9c1de7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.97:41420/0 (socket says 172.21.15.97:41420) 2024-03-25T18:16:20.619 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.614+0000 7f2fa08ce640 1 -- 172.21.15.97:0/96681940 learned_addr learned my addr 172.21.15.97:0/96681940 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:16:20.619 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.614+0000 7f2f997fa640 10 client.?.objecter ms_handle_connect 0x7f2f9c0a9920 2024-03-25T18:16:20.619 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.614+0000 7f2f997fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:20.619 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.615+0000 7f2f997fa640 10 client.?.objecter ms_handle_connect 0x7f2f9c146450 2024-03-25T18:16:20.619 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.615+0000 7f2f997fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:20.620 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.617+0000 7f2f997fa640 1 -- 172.21.15.97:0/96681940 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 299307062 0 0) 0x7f2f9c072000 con 0x7f2f9c0a9920 2024-03-25T18:16:20.621 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.617+0000 7f2f997fa640 10 monclient(hunting): _negotiate 2024-03-25T18:16:20.621 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.617+0000 7f2f997fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:20.621 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.617+0000 7f2f997fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:20.621 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.617+0000 7f2f997fa640 10 monclient(hunting): my global_id is 4156 2024-03-25T18:16:20.621 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.617+0000 7f2f997fa640 1 -- 172.21.15.97:0/96681940 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f80004eb0 con 0x7f2f9c0a9920 2024-03-25T18:16:20.621 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.617+0000 7f2f997fa640 1 -- 172.21.15.97:0/96681940 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2957788915 0 0) 0x7f2f9c13ba90 con 0x7f2f9c146450 2024-03-25T18:16:20.621 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.617+0000 7f2f997fa640 10 monclient(hunting): _negotiate 2024-03-25T18:16:20.621 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.617+0000 7f2f997fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:20.621 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.617+0000 7f2f997fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:20.621 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.617+0000 7f2f997fa640 10 monclient(hunting): my global_id is 4157 2024-03-25T18:16:20.621 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.617+0000 7f2f997fa640 1 -- 172.21.15.97:0/96681940 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f9c072000 con 0x7f2f9c146450 2024-03-25T18:16:20.628 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.626+0000 7f2f997fa640 1 -- 172.21.15.97:0/96681940 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3912223731 0 0) 0x7f2f80004eb0 con 0x7f2f9c0a9920 2024-03-25T18:16:20.628 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.626+0000 7f2f997fa640 1 -- 172.21.15.97:0/96681940 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2f9c13ba90 con 0x7f2f9c0a9920 2024-03-25T18:16:20.628 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.626+0000 7f2f997fa640 1 -- 172.21.15.97:0/96681940 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f2f90001960 con 0x7f2f9c0a9920 2024-03-25T18:16:20.628 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.626+0000 7f2f997fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:20.628 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.626+0000 7f2f997fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:16:20.628 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.626+0000 7f2f997fa640 10 monclient(hunting): dump: 2024-03-25T18:16:20.629 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2024-03-25T18:16:20.629 INFO:teuthology.orchestra.run.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:16:20.629 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:20.629 INFO:teuthology.orchestra.run.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:20.629 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:20.629 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 3 2024-03-25T18:16:20.629 INFO:teuthology.orchestra.run.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:16:20.629 INFO:teuthology.orchestra.run.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:16:20.629 INFO:teuthology.orchestra.run.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:16:20.629 INFO:teuthology.orchestra.run.smithi097.stderr: 2024-03-25T18:16:20.629 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.626+0000 7f2f997fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:16:20.629 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.626+0000 7f2f997fa640 1 -- 172.21.15.97:0/96681940 <== mon.2 v1:172.21.15.97:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3531709486 0 0) 0x7f2f9c072000 con 0x7f2f9c146450 2024-03-25T18:16:20.629 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.626+0000 7f2f997fa640 1 -- 172.21.15.97:0/96681940 --> v1:172.21.15.97:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2f80004eb0 con 0x7f2f9c146450 2024-03-25T18:16:20.629 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.626+0000 7f2f997fa640 1 -- 172.21.15.97:0/96681940 <== mon.2 v1:172.21.15.97:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f2f940013e0 con 0x7f2f9c146450 2024-03-25T18:16:20.629 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.626+0000 7f2f997fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:20.630 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.626+0000 7f2f997fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:16:20.630 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.626+0000 7f2f997fa640 10 monclient(hunting): dump: 2024-03-25T18:16:20.630 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2024-03-25T18:16:20.630 INFO:teuthology.orchestra.run.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:16:20.630 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:20.630 INFO:teuthology.orchestra.run.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:20.630 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:20.630 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 3 2024-03-25T18:16:20.630 INFO:teuthology.orchestra.run.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:16:20.630 INFO:teuthology.orchestra.run.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:16:20.630 INFO:teuthology.orchestra.run.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:16:20.630 INFO:teuthology.orchestra.run.smithi097.stderr: 2024-03-25T18:16:20.633 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.630+0000 7f2f997fa640 1 -- 172.21.15.97:0/96681940 <== mon.1 v1:172.21.15.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3607033234 0 0) 0x7f2f9c13ba90 con 0x7f2f9c0a9920 2024-03-25T18:16:20.633 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.630+0000 7f2f997fa640 1 -- 172.21.15.97:0/96681940 >> v1:172.21.15.97:6790/0 conn(0x7f2f9c146450 legacy=0x7f2f9c1e6b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:20.633 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.630+0000 7f2f997fa640 1 -- 172.21.15.97:0/96681940 >> v1:172.21.15.97:6789/0 conn(0x7f2f9c0b45a0 legacy=0x7f2f9c135fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:20.633 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.630+0000 7f2f997fa640 10 monclient: _finish_hunting 0 2024-03-25T18:16:20.633 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.630+0000 7f2f997fa640 1 monclient: found mon.b 2024-03-25T18:16:20.633 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.630+0000 7f2f997fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:20.634 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.630+0000 7f2f997fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:16:20.634 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.630+0000 7f2f997fa640 1 -- 172.21.15.97:0/96681940 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f9c1deed0 con 0x7f2f9c0a9920 2024-03-25T18:16:20.634 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.630+0000 7f2f997fa640 10 monclient: _finish_auth 0 2024-03-25T18:16:20.634 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.630+0000 7f2f997fa640 10 monclient: _check_auth_tickets 2024-03-25T18:16:20.634 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.630+0000 7f2f997fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:20.634 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.631+0000 7f2fa2b32640 5 monclient: authenticate success, global_id 4156 2024-03-25T18:16:20.634 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.631+0000 7f2fa2b32640 10 monclient: _renew_subs 2024-03-25T18:16:20.634 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.631+0000 7f2fa2b32640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:16:20.634 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.631+0000 7f2fa2b32640 1 -- 172.21.15.97:0/96681940 --> v1:172.21.15.98:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f9c13c290 con 0x7f2f9c0a9920 2024-03-25T18:16:20.634 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.631+0000 7f2fa2b32640 10 client.4156.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:16:20.634 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.631+0000 7f2fa2b32640 10 monclient: _renew_subs 2024-03-25T18:16:20.634 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.631+0000 7f2fa2b32640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:16:20.634 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.631+0000 7f2fa2b32640 1 -- 172.21.15.97:0/96681940 --> v1:172.21.15.98:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f9c138970 con 0x7f2f9c0a9920 2024-03-25T18:16:20.634 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.631+0000 7f2fa2b32640 1 librados: init done 2024-03-25T18:16:20.635 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.632+0000 7f2f86ffd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:20.635 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.633+0000 7f2f86ffd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:20.635 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.633+0000 7f2f86ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:16:20.635 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.633+0000 7f2f86ffd640 1 -- 172.21.15.97:0/96681940 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f9c1df0c0 con 0x7f2f9c0a9920 2024-03-25T18:16:20.636 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.635+0000 7f2f997fa640 1 -- 172.21.15.97:0/96681940 <== mon.1 v1:172.21.15.98:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2f900035f0 con 0x7f2f9c0a9920 2024-03-25T18:16:20.636 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.635+0000 7f2f997fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:20.637 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.635+0000 7f2f9a7fc640 4 set_mon_vals no callback set 2024-03-25T18:16:20.732 INFO:tasks.ceph.mgr.y.smithi097.stderr:2024-03-25T18:16:20.730+0000 7f7cdf810040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-25T18:16:20.789 INFO:tasks.ceph.mgr.y.smithi097.stderr:2024-03-25T18:16:20.788+0000 7f7cdf810040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-25T18:16:20.814 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.812+0000 7f2f997fa640 1 -- 172.21.15.97:0/96681940 <== mon.1 v1:172.21.15.98:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f2f90003940 con 0x7f2f9c0a9920 2024-03-25T18:16:20.814 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.812+0000 7f2f997fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:20.814 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.812+0000 7f2f997fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:16:20.814 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.812+0000 7f2f997fa640 10 monclient: dump: 2024-03-25T18:16:20.814 INFO:teuthology.orchestra.run.smithi097.stderr:epoch 1 2024-03-25T18:16:20.814 INFO:teuthology.orchestra.run.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:16:20.814 INFO:teuthology.orchestra.run.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:20.814 INFO:teuthology.orchestra.run.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:20.814 INFO:teuthology.orchestra.run.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:20.814 INFO:teuthology.orchestra.run.smithi097.stderr:election_strategy: 3 2024-03-25T18:16:20.814 INFO:teuthology.orchestra.run.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:16:20.815 INFO:teuthology.orchestra.run.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:16:20.815 INFO:teuthology.orchestra.run.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:16:20.815 INFO:teuthology.orchestra.run.smithi097.stderr: 2024-03-25T18:16:20.815 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.812+0000 7f2f997fa640 1 -- 172.21.15.97:0/96681940 <== mon.1 v1:172.21.15.98:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f2f90003eb0 con 0x7f2f9c0a9920 2024-03-25T18:16:20.815 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.812+0000 7f2f997fa640 1 -- 172.21.15.97:0/96681940 <== mon.1 v1:172.21.15.98:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 430711918 0 0) 0x7f2f90006980 con 0x7f2f9c0a9920 2024-03-25T18:16:20.815 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.812+0000 7f2f997fa640 10 client.4156.objecter ms_dispatch 0x7f2f9c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:16:20.815 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.812+0000 7f2f997fa640 3 client.4156.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:16:20.815 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.812+0000 7f2f997fa640 3 client.4156.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:16:20.815 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.812+0000 7f2f997fa640 20 client.4156.objecter dump_active .. 0 homeless 2024-03-25T18:16:20.815 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.814+0000 7f2f997fa640 1 -- 172.21.15.97:0/96681940 <== mon.1 v1:172.21.15.98:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f2f90004160 con 0x7f2f9c0a9920 2024-03-25T18:16:20.816 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.814+0000 7f2f997fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:20.816 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.814+0000 7f2f997fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:16:20.942 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.941+0000 7f2f86ffd640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:20.943 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.941+0000 7f2f86ffd640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:20.943 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.941+0000 7f2f86ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:16:20.943 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.941+0000 7f2f86ffd640 1 -- 172.21.15.97:0/96681940 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2f9c04f730 con 0x7f2f9c0a9920 2024-03-25T18:16:20.946 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:20.945+0000 7f2f997fa640 1 -- 172.21.15.97:0/96681940 <== mon.1 v1:172.21.15.98:6789/0 10 ==== mgrmap(e 2) v1 ==== 44919+0+0 (unknown 1882393413 0 0) 0x7f2f90006360 con 0x7f2f9c0a9920 2024-03-25T18:16:20.992 INFO:tasks.ceph.mgr.y.smithi097.stderr:2024-03-25T18:16:20.991+0000 7f7cdf810040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-25T18:16:21.108 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:21.107+0000 7f2f997fa640 1 -- 172.21.15.97:0/96681940 <== mon.1 v1:172.21.15.98:6789/0 11 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2169427412) 0x7f2f90004380 con 0x7f2f9c0a9920 2024-03-25T18:16:21.108 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:21.107+0000 7f2f997fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:21.108 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:21.107+0000 7f2f997fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:16:21.109 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-25T18:16:21.109 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":10,"fsid":"4c3d472c-79b8-486b-9e3d-2738d4306754","created":"2024-03-25T18:16:14.737309+0000","modified":"2024-03-25T18:16:20.357492+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:20.357492+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cb5152b9-262f-45c3-94b8-18d0e81df558","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6daf8793-0838-473d-a7d6-9a0d2894c75d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"136f1493-3b9c-4f43-a2e7-641482d69dae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"397d538c-4f64-4daf-9e8b-662d09f91327","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4f1df08-663f-4bda-93f8-3878ab7c24b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c4b18d70-a3ce-4b15-9b70-eae5525bd761","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8dd4033a-de0e-4f8a-96dc-59f3c1f825b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0077722e-ffa6-4a3d-9891-929d9869b286","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:16:21.110 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:21.108+0000 7f2f86ffd640 10 librados: watch_flush enter 2024-03-25T18:16:21.110 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:21.108+0000 7f2f86ffd640 10 librados: watch_flush exit 2024-03-25T18:16:21.110 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:21.108+0000 7f2f86ffd640 20 client.4156.objecter shutdown clearing up homeless session... 2024-03-25T18:16:21.110 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:21.108+0000 7f2f86ffd640 10 client.4156.objecter successfully canceled tick 2024-03-25T18:16:21.110 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:21.108+0000 7f2f86ffd640 10 monclient: shutdown 2024-03-25T18:16:21.110 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:21.108+0000 7f2f86ffd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:21.110 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:21.108+0000 7f2f86ffd640 1 -- 172.21.15.97:0/96681940 >> v1:172.21.15.98:6789/0 conn(0x7f2f9c0a9920 legacy=0x7f2f9c1de7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:21.110 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:21.108+0000 7f2f86ffd640 1 -- 172.21.15.97:0/96681940 shutdown_connections 2024-03-25T18:16:21.110 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:21.108+0000 7f2f86ffd640 1 -- 172.21.15.97:0/96681940 >> 172.21.15.97:0/96681940 conn(0x7f2f9c0b7f40 msgr2=0x7f2f9c0b3720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:21.110 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:21.108+0000 7f2f86ffd640 1 -- 172.21.15.97:0/96681940 shutdown_connections 2024-03-25T18:16:21.111 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:21.108+0000 7f2f86ffd640 1 -- 172.21.15.97:0/96681940 wait complete. 2024-03-25T18:16:21.111 INFO:teuthology.orchestra.run.smithi097.stderr:2024-03-25T18:16:21.108+0000 7f2f86ffd640 1 librados: shutdown 2024-03-25T18:16:21.124 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-25T18:16:21.124 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-25T18:16:21.929 DEBUG:teuthology.orchestra.run.smithi097:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:16:22.072 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:22.070+0000 7f2041d20640 10 monclient: get_monmap_and_config 2024-03-25T18:16:22.072 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:22.070+0000 7f2041d20640 10 monclient: build_initial_monmap 2024-03-25T18:16:22.072 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:22.070+0000 7f2041d20640 10 monclient: monmap: 2024-03-25T18:16:22.072 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:16:22.072 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:16:22.072 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:16:22.071970+0000 2024-03-25T18:16:22.072 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:16:22.071970+0000 2024-03-25T18:16:22.072 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:22.072 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:16:22.072 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:16:22.073 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T18:16:22.073 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-b 2024-03-25T18:16:22.073 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:16:22.074 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:22.071+0000 7f2041d20640 1 Processor -- start 2024-03-25T18:16:22.074 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:22.071+0000 7f2041d20640 1 -- start start 2024-03-25T18:16:22.074 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:22.071+0000 7f2041d20640 10 monclient: init 2024-03-25T18:16:22.075 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:22.072+0000 7f2041d20640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:22.075 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:22.072+0000 7f2041d20640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:16:22.075 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:22.073+0000 7f2041d20640 10 monclient(hunting): picked mon.noname-b con 0x7f203c0a98f0 addr v1:172.21.15.97:6790/0 2024-03-25T18:16:22.075 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:22.073+0000 7f2041d20640 10 monclient(hunting): picked mon.noname-c con 0x7f203c13f710 addr v1:172.21.15.98:6789/0 2024-03-25T18:16:22.075 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:22.073+0000 7f2041d20640 10 monclient(hunting): picked mon.noname-a con 0x7f203c1430c0 addr v1:172.21.15.97:6789/0 2024-03-25T18:16:22.075 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:22.073+0000 7f2041d20640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f203c142520 con 0x7f203c1430c0 2024-03-25T18:16:22.075 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:22.073+0000 7f2041d20640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f203c0b28a0 con 0x7f203c13f710 2024-03-25T18:16:22.075 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:22.073+0000 7f2041d20640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f203c0b2a30 con 0x7f203c0a98f0 2024-03-25T18:16:22.075 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:22.073+0000 7f2041d20640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:22.075 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:22.073+0000 7f2041d20640 10 monclient(hunting): authenticate will time out at 7594.349238s 2024-03-25T18:16:22.076 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:22.074+0000 7f203b7fe640 1 --1- >> v1:172.21.15.97:6790/0 conn(0x7f203c0a98f0 0x7f203c0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6790/0 says I am v1:172.21.15.97:59024/0 (socket says 172.21.15.97:59024) 2024-03-25T18:16:22.076 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:22.074+0000 7f203b7fe640 1 -- 172.21.15.97:0/3546974283 learned_addr learned my addr 172.21.15.97:0/3546974283 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:16:22.201 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:22.199+0000 7f203a7fc640 1 -- 172.21.15.97:0/3546974283 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3175677479 0 0) 0x7f203c142520 con 0x7f203c1430c0 2024-03-25T18:16:22.201 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:22.199+0000 7f203a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:16:22.201 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:22.199+0000 7f203a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:22.201 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:22.199+0000 7f203a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:22.201 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:22.199+0000 7f203a7fc640 10 monclient(hunting): my global_id is 4170 2024-03-25T18:16:22.202 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:22.199+0000 7f203a7fc640 1 -- 172.21.15.97:0/3546974283 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2024003520 con 0x7f203c1430c0 2024-03-25T18:16:22.208 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:22.205+0000 7f203a7fc640 1 -- 172.21.15.97:0/3546974283 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1202154682 0 0) 0x7f2024003520 con 0x7f203c1430c0 2024-03-25T18:16:22.208 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:22.205+0000 7f203a7fc640 1 -- 172.21.15.97:0/3546974283 >> v1:172.21.15.97:6790/0 conn(0x7f203c0a98f0 legacy=0x7f203c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:22.208 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:22.205+0000 7f203a7fc640 1 -- 172.21.15.97:0/3546974283 >> v1:172.21.15.98:6789/0 conn(0x7f203c13f710 legacy=0x7f203c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:16:22.208 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:22.205+0000 7f203a7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:16:22.208 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:22.205+0000 7f203a7fc640 1 monclient: found mon.noname-a 2024-03-25T18:16:22.208 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:22.205+0000 7f203a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:22.208 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:22.205+0000 7f203a7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.97:6789/0 2024-03-25T18:16:22.208 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:22.205+0000 7f203a7fc640 1 -- 172.21.15.97:0/3546974283 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f203c0b44c0 con 0x7f203c1430c0 2024-03-25T18:16:22.208 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:22.205+0000 7f203a7fc640 10 monclient: _finish_auth 0 2024-03-25T18:16:22.208 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:22.205+0000 7f203a7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:16:22.208 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:22.205+0000 7f203a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:22.208 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:22.205+0000 7f2041d20640 5 monclient: authenticate success, global_id 4170 2024-03-25T18:16:22.208 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:22.205+0000 7f2041d20640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:22.209 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:22.205+0000 7f203a7fc640 1 -- 172.21.15.97:0/3546974283 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f2034001a10 con 0x7f203c1430c0 2024-03-25T18:16:22.209 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:22.205+0000 7f203a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:22.209 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:22.205+0000 7f203a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:16:22.209 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:22.205+0000 7f203a7fc640 10 monclient: dump: 2024-03-25T18:16:22.209 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:16:22.209 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:16:22.209 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:22.209 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:22.209 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:22.209 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:16:22.209 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:16:22.210 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:16:22.210 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:16:22.210 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:16:22.210 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:22.205+0000 7f2041d20640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:22.215 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:22.214+0000 7f203a7fc640 1 -- 172.21.15.97:0/3546974283 <== mon.0 v1:172.21.15.97:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2034001ca0 con 0x7f203c1430c0 2024-03-25T18:16:22.215 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:22.214+0000 7f203a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:22.215 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:22.214+0000 7f2041d20640 10 monclient: get_monmap_and_config success 2024-03-25T18:16:22.216 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:22.214+0000 7f2041d20640 4 set_mon_vals no callback set 2024-03-25T18:16:22.216 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:22.215+0000 7f2041d20640 10 monclient: shutdown 2024-03-25T18:16:22.217 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:22.215+0000 7f2041d20640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:22.217 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:22.215+0000 7f2041d20640 1 -- 172.21.15.97:0/3546974283 >> v1:172.21.15.97:6789/0 conn(0x7f203c1430c0 legacy=0x7f203c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:22.217 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:22.215+0000 7f2041d20640 1 -- 172.21.15.97:0/3546974283 shutdown_connections 2024-03-25T18:16:22.217 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:22.215+0000 7f2041d20640 1 -- 172.21.15.97:0/3546974283 >> 172.21.15.97:0/3546974283 conn(0x7f203c0b7ea0 msgr2=0x7f203c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:22.217 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:22.215+0000 7f2041d20640 1 -- 172.21.15.97:0/3546974283 shutdown_connections 2024-03-25T18:16:22.217 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:22.215+0000 7f2041d20640 1 -- 172.21.15.97:0/3546974283 wait complete. 2024-03-25T18:16:22.218 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:22.217+0000 7f2041d20640 10 monclient: build_initial_monmap 2024-03-25T18:16:22.218 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:22.217+0000 7f2041d20640 10 monclient: monmap: 2024-03-25T18:16:22.218 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:16:22.218 INFO:teuthology.misc.health.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:16:22.218 INFO:teuthology.misc.health.smithi097.stderr:last_changed 0.000000 2024-03-25T18:16:22.218 INFO:teuthology.misc.health.smithi097.stderr:created 0.000000 2024-03-25T18:16:22.218 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:22.218 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 1 2024-03-25T18:16:22.218 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:16:22.218 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T18:16:22.218 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-c 2024-03-25T18:16:22.218 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:16:22.219 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:22.217+0000 7f2041d20640 1 librados: starting msgr at 2024-03-25T18:16:22.219 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:22.217+0000 7f2041d20640 1 librados: starting objecter 2024-03-25T18:16:22.220 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:22.218+0000 7f2041d20640 1 Processor -- start 2024-03-25T18:16:22.220 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:22.218+0000 7f2041d20640 1 -- start start 2024-03-25T18:16:22.220 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:22.218+0000 7f2041d20640 1 librados: setting wanted keys 2024-03-25T18:16:22.220 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:22.218+0000 7f2041d20640 1 librados: calling monclient init 2024-03-25T18:16:22.220 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:22.218+0000 7f2041d20640 10 monclient: init 2024-03-25T18:16:22.221 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:22.219+0000 7f2041d20640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:22.221 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:22.219+0000 7f2041d20640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:16:22.221 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:22.220+0000 7f2041d20640 10 monclient(hunting): picked mon.noname-b con 0x7f203c0a98f0 addr v1:172.21.15.98:6789/0 2024-03-25T18:16:22.221 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:22.220+0000 7f2041d20640 10 monclient(hunting): picked mon.noname-a con 0x7f203c13f710 addr v1:172.21.15.97:6789/0 2024-03-25T18:16:22.222 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:22.220+0000 7f2041d20640 10 monclient(hunting): picked mon.noname-c con 0x7f203c1430c0 addr v1:172.21.15.97:6790/0 2024-03-25T18:16:22.222 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:22.220+0000 7f2041d20640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f203c138520 con 0x7f203c13f710 2024-03-25T18:16:22.222 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:22.220+0000 7f2041d20640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f203c138770 con 0x7f203c0a98f0 2024-03-25T18:16:22.222 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:22.220+0000 7f2041d20640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f203c1393a0 con 0x7f203c1430c0 2024-03-25T18:16:22.222 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:22.220+0000 7f2041d20640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:22.222 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:22.220+0000 7f2041d20640 10 monclient(hunting): authenticate will time out at 7594.496495s 2024-03-25T18:16:22.269 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:22.268+0000 7f203affd640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f203c13f710 0x7f203c1d4920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:57674/0 (socket says 172.21.15.97:57674) 2024-03-25T18:16:22.270 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:22.268+0000 7f203affd640 1 -- 172.21.15.97:0/4034841291 learned_addr learned my addr 172.21.15.97:0/4034841291 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:16:22.271 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:22.270+0000 7f2023fff640 10 client.?.objecter ms_handle_connect 0x7f203c13f710 2024-03-25T18:16:22.271 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:22.270+0000 7f2023fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:22.275 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:22.273+0000 7f2023fff640 1 -- 172.21.15.97:0/4034841291 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1649327848 0 0) 0x7f203c138520 con 0x7f203c13f710 2024-03-25T18:16:22.275 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:22.273+0000 7f2023fff640 10 monclient(hunting): _negotiate 2024-03-25T18:16:22.275 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:22.273+0000 7f2023fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:22.275 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:22.273+0000 7f2023fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:22.275 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:22.273+0000 7f2023fff640 10 monclient(hunting): my global_id is 4173 2024-03-25T18:16:22.275 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:22.273+0000 7f2023fff640 1 -- 172.21.15.97:0/4034841291 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2028004eb0 con 0x7f203c13f710 2024-03-25T18:16:22.299 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:22.297+0000 7f2023fff640 1 -- 172.21.15.97:0/4034841291 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 724115639 0 0) 0x7f2028004eb0 con 0x7f203c13f710 2024-03-25T18:16:22.299 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:22.298+0000 7f2023fff640 1 -- 172.21.15.97:0/4034841291 --> v1:172.21.15.97:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f203c138520 con 0x7f203c13f710 2024-03-25T18:16:22.299 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:22.298+0000 7f2023fff640 1 -- 172.21.15.97:0/4034841291 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f2030001960 con 0x7f203c13f710 2024-03-25T18:16:22.299 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:22.298+0000 7f2023fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:22.299 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:22.298+0000 7f2023fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:16:22.300 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:22.298+0000 7f2023fff640 10 monclient(hunting): dump: 2024-03-25T18:16:22.300 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:16:22.300 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:16:22.300 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:22.300 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:22.300 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:22.300 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:16:22.300 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:16:22.300 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:16:22.300 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:16:22.300 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:16:22.300 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:22.298+0000 7f2023fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:16:22.307 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:22.303+0000 7f2023fff640 1 -- 172.21.15.97:0/4034841291 <== mon.0 v1:172.21.15.97:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1187439946 0 0) 0x7f203c138520 con 0x7f203c13f710 2024-03-25T18:16:22.307 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:22.303+0000 7f2023fff640 1 -- 172.21.15.97:0/4034841291 >> v1:172.21.15.97:6790/0 conn(0x7f203c1430c0 legacy=0x7f203c1dade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:22.307 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:22.303+0000 7f2023fff640 1 -- 172.21.15.97:0/4034841291 >> v1:172.21.15.98:6789/0 conn(0x7f203c0a98f0 legacy=0x7f203c138c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:22.307 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:22.303+0000 7f2023fff640 10 monclient: _finish_hunting 0 2024-03-25T18:16:22.307 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:22.303+0000 7f2023fff640 1 monclient: found mon.a 2024-03-25T18:16:22.307 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:22.303+0000 7f2023fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:22.307 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:22.303+0000 7f2023fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:16:22.307 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:22.303+0000 7f2023fff640 1 -- 172.21.15.97:0/4034841291 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f203c139530 con 0x7f203c13f710 2024-03-25T18:16:22.307 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:22.303+0000 7f2023fff640 10 monclient: _finish_auth 0 2024-03-25T18:16:22.307 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:22.303+0000 7f2023fff640 10 monclient: _check_auth_tickets 2024-03-25T18:16:22.307 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:22.303+0000 7f2023fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:22.307 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:22.304+0000 7f2041d20640 5 monclient: authenticate success, global_id 4173 2024-03-25T18:16:22.307 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:22.304+0000 7f2041d20640 10 monclient: _renew_subs 2024-03-25T18:16:22.308 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:22.304+0000 7f2041d20640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:16:22.308 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:22.304+0000 7f2041d20640 1 -- 172.21.15.97:0/4034841291 --> v1:172.21.15.97:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f203c1d5030 con 0x7f203c13f710 2024-03-25T18:16:22.309 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:22.304+0000 7f2041d20640 10 client.4173.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:16:22.309 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:22.304+0000 7f2041d20640 10 monclient: _renew_subs 2024-03-25T18:16:22.309 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:22.304+0000 7f2041d20640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:16:22.309 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:22.304+0000 7f2041d20640 1 -- 172.21.15.97:0/4034841291 --> v1:172.21.15.97:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f203c1d52e0 con 0x7f203c13f710 2024-03-25T18:16:22.309 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:22.304+0000 7f2041d20640 1 librados: init done 2024-03-25T18:16:22.309 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:22.305+0000 7f2041d20640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:22.310 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:22.305+0000 7f2041d20640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:22.310 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:22.305+0000 7f2041d20640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:16:22.310 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:22.305+0000 7f2041d20640 1 -- 172.21.15.97:0/4034841291 --> v1:172.21.15.97:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f203c136c70 con 0x7f203c13f710 2024-03-25T18:16:22.400 INFO:tasks.ceph.mgr.x.smithi098.stderr:2024-03-25T18:16:22.399+0000 7f1608637640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-25T18:16:22.400 INFO:tasks.ceph.mgr.x.smithi098.stderr:2024-03-25T18:16:22.399+0000 7f1608637640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-25T18:16:22.528 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:22.527+0000 7f2023fff640 1 -- 172.21.15.97:0/4034841291 <== mon.0 v1:172.21.15.97:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2030001ac0 con 0x7f203c13f710 2024-03-25T18:16:22.528 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:22.527+0000 7f2023fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:22.529 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:22.527+0000 7f2039ffb640 4 set_mon_vals no callback set 2024-03-25T18:16:22.661 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:22.660+0000 7f2023fff640 1 -- 172.21.15.97:0/4034841291 <== mon.0 v1:172.21.15.97:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f2030001d80 con 0x7f203c13f710 2024-03-25T18:16:22.662 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:22.660+0000 7f2023fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:22.662 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:22.660+0000 7f2023fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:16:22.662 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:22.660+0000 7f2023fff640 10 monclient: dump: 2024-03-25T18:16:22.662 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:16:22.662 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:16:22.662 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:22.662 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:22.662 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:22.662 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:16:22.662 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:16:22.662 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:16:22.662 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:16:22.663 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:16:22.663 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:22.661+0000 7f2023fff640 1 -- 172.21.15.97:0/4034841291 <== mon.0 v1:172.21.15.97:6789/0 7 ==== mgrmap(e 3) v1 ==== 89591+0+0 (unknown 497454952 0 0) 0x7f2030003920 con 0x7f203c13f710 2024-03-25T18:16:22.663 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:22.662+0000 7f203b7fe640 10 monclient: get_auth_request con 0x7f202806d9b0 auth_method 0 2024-03-25T18:16:22.663 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:22.662+0000 7f2023fff640 10 client.4173.objecter ms_handle_connect 0x7f202806d9b0 2024-03-25T18:16:22.668 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:22.667+0000 7f2023fff640 1 -- 172.21.15.97:0/4034841291 <== mon.0 v1:172.21.15.97:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 430711918 0 0) 0x7f2030054520 con 0x7f203c13f710 2024-03-25T18:16:22.668 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:22.667+0000 7f2023fff640 10 client.4173.objecter ms_dispatch 0x7f203c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:16:22.669 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:22.667+0000 7f2023fff640 3 client.4173.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:16:22.669 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:22.667+0000 7f2023fff640 3 client.4173.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:16:22.669 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:22.667+0000 7f2023fff640 20 client.4173.objecter dump_active .. 0 homeless 2024-03-25T18:16:22.669 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:22.667+0000 7f2023fff640 1 -- 172.21.15.97:0/4034841291 <== mon.0 v1:172.21.15.97:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f203004fe60 con 0x7f203c13f710 2024-03-25T18:16:22.669 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:22.667+0000 7f2023fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:22.669 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:22.667+0000 7f2023fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:16:23.017 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:23.016+0000 7f2041d20640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:23.018 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:23.016+0000 7f2041d20640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:23.018 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:23.016+0000 7f2041d20640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:16:23.018 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:23.016+0000 7f2041d20640 1 -- 172.21.15.97:0/4034841291 --> v1:172.21.15.97:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f203c139530 con 0x7f203c13f710 2024-03-25T18:16:23.078 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:23.077+0000 7f2023fff640 1 -- 172.21.15.97:0/4034841291 <== mon.0 v1:172.21.15.97:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2169427412) 0x7f203c139530 con 0x7f203c13f710 2024-03-25T18:16:23.078 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:23.077+0000 7f2023fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:23.078 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:23.077+0000 7f2023fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:16:23.078 INFO:teuthology.misc.health.smithi097.stdout: 2024-03-25T18:16:23.079 INFO:teuthology.misc.health.smithi097.stdout:{"epoch":10,"fsid":"4c3d472c-79b8-486b-9e3d-2738d4306754","created":"2024-03-25T18:16:14.737309+0000","modified":"2024-03-25T18:16:20.357492+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:20.357492+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cb5152b9-262f-45c3-94b8-18d0e81df558","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6daf8793-0838-473d-a7d6-9a0d2894c75d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"136f1493-3b9c-4f43-a2e7-641482d69dae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"397d538c-4f64-4daf-9e8b-662d09f91327","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4f1df08-663f-4bda-93f8-3878ab7c24b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c4b18d70-a3ce-4b15-9b70-eae5525bd761","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8dd4033a-de0e-4f8a-96dc-59f3c1f825b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0077722e-ffa6-4a3d-9891-929d9869b286","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:16:23.081 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:23.079+0000 7f2041d20640 10 librados: watch_flush enter 2024-03-25T18:16:23.081 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:23.079+0000 7f2041d20640 10 librados: watch_flush exit 2024-03-25T18:16:23.081 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:23.079+0000 7f2041d20640 20 client.4173.objecter shutdown clearing up homeless session... 2024-03-25T18:16:23.081 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:23.079+0000 7f2041d20640 10 client.4173.objecter successfully canceled tick 2024-03-25T18:16:23.081 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:23.079+0000 7f2041d20640 1 -- 172.21.15.97:0/4034841291 >> v1:172.21.15.98:6800/800387573 conn(0x7f202806d9b0 legacy=0x7f202806fe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:23.082 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:23.079+0000 7f2041d20640 10 monclient: shutdown 2024-03-25T18:16:23.082 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:23.079+0000 7f2041d20640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:23.082 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:23.079+0000 7f2041d20640 1 -- 172.21.15.97:0/4034841291 >> v1:172.21.15.97:6789/0 conn(0x7f203c13f710 legacy=0x7f203c1d4920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:23.082 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:23.080+0000 7f2041d20640 1 -- 172.21.15.97:0/4034841291 shutdown_connections 2024-03-25T18:16:23.082 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:23.080+0000 7f2041d20640 1 -- 172.21.15.97:0/4034841291 >> 172.21.15.97:0/4034841291 conn(0x7f203c0b7ea0 msgr2=0x7f203c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:23.082 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:23.080+0000 7f2041d20640 1 -- 172.21.15.97:0/4034841291 shutdown_connections 2024-03-25T18:16:23.082 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:23.080+0000 7f2041d20640 1 -- 172.21.15.97:0/4034841291 wait complete. 2024-03-25T18:16:23.082 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:23.080+0000 7f2041d20640 1 librados: shutdown 2024-03-25T18:16:23.091 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:16:27.340 INFO:tasks.ceph.osd.7.smithi098.stderr:==00:00:00:06.819 35255== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:27.396 INFO:tasks.ceph.osd.6.smithi098.stderr:==00:00:00:06.894 35254== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:27.401 INFO:tasks.ceph.osd.7.smithi098.stderr:==00:00:00:06.880 35255== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:27.457 INFO:tasks.ceph.osd.6.smithi098.stderr:==00:00:00:06.955 35254== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:27.477 INFO:tasks.ceph.osd.4.smithi098.stderr:==00:00:00:06.984 35252== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:27.532 INFO:tasks.ceph.osd.5.smithi098.stderr:==00:00:00:07.037 35253== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:27.536 INFO:tasks.ceph.osd.4.smithi098.stderr:==00:00:00:07.043 35252== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:27.591 INFO:tasks.ceph.osd.5.smithi098.stderr:==00:00:00:07.096 35253== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:27.646 INFO:tasks.ceph.osd.0.smithi097.stderr:==00:00:00:07.154 35245== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:27.708 INFO:tasks.ceph.osd.0.smithi097.stderr:==00:00:00:07.216 35245== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:27.725 INFO:tasks.ceph.osd.2.smithi097.stderr:==00:00:00:07.225 35247== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:27.784 INFO:tasks.ceph.osd.2.smithi097.stderr:==00:00:00:07.285 35247== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:27.954 INFO:tasks.ceph.osd.7.smithi098.stderr:2024-03-25T18:16:27.945+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:16:28.016 INFO:tasks.ceph.osd.6.smithi098.stderr:2024-03-25T18:16:28.009+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:16:28.085 INFO:tasks.ceph.osd.4.smithi098.stderr:2024-03-25T18:16:28.079+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:16:28.202 INFO:tasks.ceph.osd.5.smithi098.stderr:2024-03-25T18:16:28.193+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:16:28.280 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-03-25T18:16:28.271+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:16:28.340 INFO:tasks.ceph.osd.2.smithi097.stderr:2024-03-25T18:16:28.333+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:16:28.416 INFO:tasks.ceph.osd.1.smithi097.stderr:==00:00:00:07.905 35248== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:28.436 INFO:tasks.ceph.osd.3.smithi097.stderr:==00:00:00:07.943 35246== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:28.475 INFO:tasks.ceph.osd.1.smithi097.stderr:==00:00:00:07.964 35248== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:28.490 INFO:tasks.ceph.osd.6.smithi098.stderr:==00:00:00:07.988 35254== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:28.495 INFO:tasks.ceph.osd.3.smithi097.stderr:==00:00:00:08.003 35246== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:28.510 INFO:tasks.ceph.osd.7.smithi098.stderr:==00:00:00:07.989 35255== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:28.557 INFO:tasks.ceph.osd.4.smithi098.stderr:==00:00:00:08.064 35252== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:28.712 INFO:tasks.ceph.osd.5.smithi098.stderr:==00:00:00:08.217 35253== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:28.740 INFO:tasks.ceph.osd.0.smithi097.stderr:==00:00:00:08.248 35245== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:28.821 INFO:tasks.ceph.osd.2.smithi097.stderr:==00:00:00:08.322 35247== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:28.859 INFO:tasks.ceph.osd.6.smithi098.stderr:==00:00:00:08.357 35254== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:28.881 INFO:tasks.ceph.osd.7.smithi098.stderr:==00:00:00:08.360 35255== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:28.926 INFO:tasks.ceph.osd.4.smithi098.stderr:==00:00:00:08.433 35252== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:29.021 INFO:tasks.ceph.osd.1.smithi097.stderr:2024-03-25T18:16:29.014+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:16:29.064 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-03-25T18:16:29.055+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:16:29.073 INFO:tasks.ceph.osd.5.smithi098.stderr:==00:00:00:08.578 35253== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:29.121 INFO:tasks.ceph.osd.0.smithi097.stderr:==00:00:00:08.628 35245== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:29.176 INFO:tasks.ceph.osd.6.smithi098.stderr:==00:00:00:08.674 35254== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:29.184 INFO:tasks.ceph.osd.0.smithi097.stderr:==00:00:00:08.692 35245== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:29.190 INFO:tasks.ceph.osd.2.smithi097.stderr:==00:00:00:08.691 35247== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:29.199 INFO:tasks.ceph.osd.7.smithi098.stderr:==00:00:00:08.678 35255== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:29.278 INFO:tasks.ceph.osd.4.smithi098.stderr:==00:00:00:08.784 35252== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:29.383 INFO:tasks.ceph.osd.5.smithi098.stderr:==00:00:00:08.888 35253== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:29.446 INFO:tasks.ceph.osd.6.smithi098.stderr:==00:00:00:08.944 35254== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:29.449 INFO:tasks.ceph.osd.0.smithi097.stderr:==00:00:00:08.957 35245== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:29.474 INFO:tasks.ceph.osd.7.smithi098.stderr:==00:00:00:08.953 35255== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:29.480 INFO:tasks.ceph.osd.1.smithi097.stderr:==00:00:00:08.969 35248== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:29.505 INFO:tasks.ceph.osd.2.smithi097.stderr:==00:00:00:09.005 35247== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:29.520 INFO:tasks.ceph.osd.3.smithi097.stderr:==00:00:00:09.027 35246== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:29.565 INFO:tasks.ceph.osd.4.smithi098.stderr:==00:00:00:09.071 35252== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:29.648 INFO:tasks.ceph.osd.5.smithi098.stderr:==00:00:00:09.153 35253== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:29.710 INFO:tasks.ceph.osd.6.smithi098.stderr:==00:00:00:09.208 35254== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:29.718 INFO:tasks.ceph.osd.0.smithi097.stderr:==00:00:00:09.225 35245== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:29.738 INFO:tasks.ceph.osd.7.smithi098.stderr:==00:00:00:09.217 35255== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:29.782 INFO:tasks.ceph.osd.2.smithi097.stderr:==00:00:00:09.283 35247== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:29.852 INFO:tasks.ceph.osd.1.smithi097.stderr:==00:00:00:09.341 35248== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:29.852 INFO:tasks.ceph.osd.4.smithi098.stderr:==00:00:00:09.359 35252== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:29.891 INFO:tasks.ceph.osd.3.smithi097.stderr:==00:00:00:09.399 35246== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:29.896 DEBUG:teuthology.orchestra.run.smithi097:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:16:29.916 INFO:tasks.ceph.osd.5.smithi098.stderr:==00:00:00:09.421 35253== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:29.976 INFO:tasks.ceph.osd.6.smithi098.stderr:==00:00:00:09.474 35254== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:29.984 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:29.983+0000 7f2af1748640 10 monclient: get_monmap_and_config 2024-03-25T18:16:29.984 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:29.983+0000 7f2af1748640 10 monclient: build_initial_monmap 2024-03-25T18:16:29.984 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:29.983+0000 7f2af1748640 10 monclient: monmap: 2024-03-25T18:16:29.984 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:16:29.984 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:16:29.984 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:16:29.984608+0000 2024-03-25T18:16:29.984 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:16:29.984608+0000 2024-03-25T18:16:29.985 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:29.985 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:16:29.985 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:16:29.985 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T18:16:29.985 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-b 2024-03-25T18:16:29.985 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:16:29.986 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:29.983+0000 7f2af1748640 1 Processor -- start 2024-03-25T18:16:29.986 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:29.983+0000 7f2af1748640 1 -- start start 2024-03-25T18:16:29.986 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:29.983+0000 7f2af1748640 10 monclient: init 2024-03-25T18:16:29.986 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:29.984+0000 7f2af1748640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:29.986 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:29.984+0000 7f2af1748640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:16:29.986 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:29.984+0000 7f2af1748640 10 monclient(hunting): picked mon.noname-a con 0x7f2aec0b10d0 addr v1:172.21.15.97:6789/0 2024-03-25T18:16:29.987 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:29.984+0000 7f2af1748640 10 monclient(hunting): picked mon.noname-b con 0x7f2aec0b4360 addr v1:172.21.15.97:6790/0 2024-03-25T18:16:29.987 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:29.984+0000 7f2af1748640 10 monclient(hunting): picked mon.noname-c con 0x7f2aec0a9470 addr v1:172.21.15.98:6789/0 2024-03-25T18:16:29.987 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:29.984+0000 7f2af1748640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2aec0a9ab0 con 0x7f2aec0b10d0 2024-03-25T18:16:29.987 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:29.984+0000 7f2af1748640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2aec0b2400 con 0x7f2aec0a9470 2024-03-25T18:16:29.987 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:29.984+0000 7f2af1748640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2aec0aa2a0 con 0x7f2aec0b4360 2024-03-25T18:16:29.987 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:29.984+0000 7f2af1748640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:29.987 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:29.984+0000 7f2af1748640 10 monclient(hunting): authenticate will time out at 7602.260470s 2024-03-25T18:16:29.987 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:29.985+0000 7f2aea7fc640 1 --1- >> v1:172.21.15.97:6790/0 conn(0x7f2aec0b4360 0x7f2aec0b1cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6790/0 says I am v1:172.21.15.97:59080/0 (socket says 172.21.15.97:59080) 2024-03-25T18:16:29.987 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:29.985+0000 7f2aea7fc640 1 -- 172.21.15.97:0/1922048496 learned_addr learned my addr 172.21.15.97:0/1922048496 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:16:29.988 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:29.987+0000 7f2ae9ffb640 1 -- 172.21.15.97:0/1922048496 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1874075414 0 0) 0x7f2aec0b2400 con 0x7f2aec0a9470 2024-03-25T18:16:29.988 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:29.987+0000 7f2ae9ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:16:29.989 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:29.987+0000 7f2ae9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:29.989 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:29.987+0000 7f2ae9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:29.989 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:29.987+0000 7f2ae9ffb640 10 monclient(hunting): my global_id is 4198 2024-03-25T18:16:29.989 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:29.987+0000 7f2ae9ffb640 1 -- 172.21.15.97:0/1922048496 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ad4003520 con 0x7f2aec0a9470 2024-03-25T18:16:29.989 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:29.987+0000 7f2ae9ffb640 1 -- 172.21.15.97:0/1922048496 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1688199165 0 0) 0x7f2aec0aa2a0 con 0x7f2aec0b4360 2024-03-25T18:16:29.989 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:29.987+0000 7f2ae9ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:16:29.989 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:29.987+0000 7f2ae9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:29.989 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:29.987+0000 7f2ae9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:29.989 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:29.987+0000 7f2ae9ffb640 10 monclient(hunting): my global_id is 4193 2024-03-25T18:16:29.989 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:29.987+0000 7f2ae9ffb640 1 -- 172.21.15.97:0/1922048496 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2aec0b2400 con 0x7f2aec0b4360 2024-03-25T18:16:29.989 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:29.987+0000 7f2ae9ffb640 1 -- 172.21.15.97:0/1922048496 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2504000369 0 0) 0x7f2aec0a9ab0 con 0x7f2aec0b10d0 2024-03-25T18:16:29.989 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:29.987+0000 7f2ae9ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:16:29.989 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:29.987+0000 7f2ae9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:29.989 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:29.987+0000 7f2ae9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:29.990 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:29.987+0000 7f2ae9ffb640 10 monclient(hunting): my global_id is 4200 2024-03-25T18:16:29.990 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:29.987+0000 7f2ae9ffb640 1 -- 172.21.15.97:0/1922048496 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2aec0aa2a0 con 0x7f2aec0b10d0 2024-03-25T18:16:29.990 INFO:tasks.ceph.osd.0.smithi097.stderr:==00:00:00:09.498 35245== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:29.992 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:29.990+0000 7f2ae9ffb640 1 -- 172.21.15.97:0/1922048496 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1754294981 0 0) 0x7f2aec0aa2a0 con 0x7f2aec0b10d0 2024-03-25T18:16:29.992 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:29.990+0000 7f2ae9ffb640 1 -- 172.21.15.97:0/1922048496 >> v1:172.21.15.97:6790/0 conn(0x7f2aec0b4360 legacy=0x7f2aec0b1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:29.992 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:29.990+0000 7f2ae9ffb640 1 -- 172.21.15.97:0/1922048496 >> v1:172.21.15.98:6789/0 conn(0x7f2aec0a9470 legacy=0x7f2aec14dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:29.992 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:29.990+0000 7f2ae9ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:16:29.992 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:29.990+0000 7f2ae9ffb640 1 monclient: found mon.noname-a 2024-03-25T18:16:29.992 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:29.990+0000 7f2ae9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:29.992 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:29.990+0000 7f2ae9ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.97:6789/0 2024-03-25T18:16:29.992 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:29.990+0000 7f2ae9ffb640 1 -- 172.21.15.97:0/1922048496 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2aec0b14b0 con 0x7f2aec0b10d0 2024-03-25T18:16:29.992 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:29.990+0000 7f2ae9ffb640 10 monclient: _finish_auth 0 2024-03-25T18:16:29.992 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:29.990+0000 7f2ae9ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:16:29.992 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:29.990+0000 7f2ae9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:29.992 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:29.990+0000 7f2ae9ffb640 1 -- 172.21.15.97:0/1922048496 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f2ae40015b0 con 0x7f2aec0b10d0 2024-03-25T18:16:29.992 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:29.990+0000 7f2ae9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:29.992 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:29.990+0000 7f2ae9ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:16:29.993 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:29.990+0000 7f2ae9ffb640 10 monclient: dump: 2024-03-25T18:16:29.993 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:16:29.993 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:16:29.993 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:29.993 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:29.993 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:29.993 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:16:29.993 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:16:29.993 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:16:29.993 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:16:29.993 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:16:29.993 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:29.990+0000 7f2af1748640 5 monclient: authenticate success, global_id 4200 2024-03-25T18:16:29.993 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:29.990+0000 7f2af1748640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:29.994 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:29.992+0000 7f2ae9ffb640 1 -- 172.21.15.97:0/1922048496 <== mon.0 v1:172.21.15.97:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2ae4001710 con 0x7f2aec0b10d0 2024-03-25T18:16:29.995 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:29.992+0000 7f2ae9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:29.995 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:29.992+0000 7f2ae9ffb640 1 -- 172.21.15.97:0/1922048496 <== mon.0 v1:172.21.15.97:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f2ae4001b60 con 0x7f2aec0b10d0 2024-03-25T18:16:29.995 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:29.992+0000 7f2ae9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:29.995 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:29.992+0000 7f2ae9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:16:29.995 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:29.992+0000 7f2ae9ffb640 10 monclient: dump: 2024-03-25T18:16:29.995 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:16:29.995 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:16:29.995 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:29.995 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:29.995 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:29.995 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:16:29.995 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:16:29.995 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:16:29.995 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:16:29.996 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:16:29.996 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:29.992+0000 7f2af1748640 10 monclient: get_monmap_and_config success 2024-03-25T18:16:29.996 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:29.992+0000 7f2af1748640 4 set_mon_vals no callback set 2024-03-25T18:16:29.996 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:29.993+0000 7f2af1748640 10 monclient: shutdown 2024-03-25T18:16:29.996 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:29.993+0000 7f2af1748640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:29.996 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:29.993+0000 7f2af1748640 1 -- 172.21.15.97:0/1922048496 >> v1:172.21.15.97:6789/0 conn(0x7f2aec0b10d0 legacy=0x7f2aec0b4ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:29.996 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:29.993+0000 7f2af1748640 1 -- 172.21.15.97:0/1922048496 shutdown_connections 2024-03-25T18:16:29.996 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:29.993+0000 7f2af1748640 1 -- 172.21.15.97:0/1922048496 >> 172.21.15.97:0/1922048496 conn(0x7f2aec0617a0 msgr2=0x7f2aec061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:29.996 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:29.993+0000 7f2af1748640 1 -- 172.21.15.97:0/1922048496 shutdown_connections 2024-03-25T18:16:29.996 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:29.993+0000 7f2af1748640 1 -- 172.21.15.97:0/1922048496 wait complete. 2024-03-25T18:16:29.996 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:29.994+0000 7f2af1748640 10 monclient: build_initial_monmap 2024-03-25T18:16:29.996 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:29.994+0000 7f2af1748640 10 monclient: monmap: 2024-03-25T18:16:29.996 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:16:29.996 INFO:teuthology.misc.health.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:16:29.996 INFO:teuthology.misc.health.smithi097.stderr:last_changed 0.000000 2024-03-25T18:16:29.997 INFO:teuthology.misc.health.smithi097.stderr:created 0.000000 2024-03-25T18:16:29.997 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:29.997 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 1 2024-03-25T18:16:29.997 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:16:29.997 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T18:16:29.997 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-c 2024-03-25T18:16:29.997 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:16:29.999 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:29.994+0000 7f2af1748640 1 librados: starting msgr at 2024-03-25T18:16:29.999 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:29.994+0000 7f2af1748640 1 librados: starting objecter 2024-03-25T18:16:29.999 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:29.994+0000 7f2af1748640 1 Processor -- start 2024-03-25T18:16:29.999 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:29.994+0000 7f2af1748640 1 -- start start 2024-03-25T18:16:29.999 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:29.994+0000 7f2af1748640 1 librados: setting wanted keys 2024-03-25T18:16:30.000 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:29.994+0000 7f2af1748640 1 librados: calling monclient init 2024-03-25T18:16:30.000 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:29.994+0000 7f2af1748640 10 monclient: init 2024-03-25T18:16:30.000 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:29.995+0000 7f2af1748640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:30.000 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:29.995+0000 7f2af1748640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:16:30.000 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:29.995+0000 7f2af1748640 10 monclient(hunting): picked mon.noname-a con 0x7f2aec0a9470 addr v1:172.21.15.97:6789/0 2024-03-25T18:16:30.000 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:29.995+0000 7f2af1748640 10 monclient(hunting): picked mon.noname-c con 0x7f2aec0b10d0 addr v1:172.21.15.97:6790/0 2024-03-25T18:16:30.000 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:29.995+0000 7f2af1748640 10 monclient(hunting): picked mon.noname-b con 0x7f2aec0b4360 addr v1:172.21.15.98:6789/0 2024-03-25T18:16:30.000 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:29.995+0000 7f2af1748640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2aec0ac140 con 0x7f2aec0a9470 2024-03-25T18:16:30.000 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:29.995+0000 7f2af1748640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2aec1d80c0 con 0x7f2aec0b4360 2024-03-25T18:16:30.000 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:29.995+0000 7f2af1748640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2aec1d7490 con 0x7f2aec0b10d0 2024-03-25T18:16:30.000 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:29.995+0000 7f2af1748640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:30.000 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:29.995+0000 7f2af1748640 10 monclient(hunting): authenticate will time out at 7602.271253s 2024-03-25T18:16:30.000 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:29.996+0000 7f2aeb7fe640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f2aec0b4360 0x7f2aec1331c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.97:41508/0 (socket says 172.21.15.97:41508) 2024-03-25T18:16:30.000 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:29.996+0000 7f2aeb7fe640 1 -- 172.21.15.97:0/1633116660 learned_addr learned my addr 172.21.15.97:0/1633116660 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:16:30.001 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:29.996+0000 7f2ad37fe640 10 client.?.objecter ms_handle_connect 0x7f2aec0b4360 2024-03-25T18:16:30.001 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:29.996+0000 7f2ad37fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:30.001 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:29.996+0000 7f2ad37fe640 10 client.?.objecter ms_handle_connect 0x7f2aec0b10d0 2024-03-25T18:16:30.001 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:29.996+0000 7f2ad37fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:30.001 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:29.996+0000 7f2ad37fe640 10 client.?.objecter ms_handle_connect 0x7f2aec0a9470 2024-03-25T18:16:30.001 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:29.996+0000 7f2ad37fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:30.001 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:29.997+0000 7f2ad37fe640 1 -- 172.21.15.97:0/1633116660 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3882107140 0 0) 0x7f2aec1d7490 con 0x7f2aec0b10d0 2024-03-25T18:16:30.001 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:29.998+0000 7f2ad37fe640 10 monclient(hunting): _negotiate 2024-03-25T18:16:30.001 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:29.998+0000 7f2ad37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:30.001 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:29.998+0000 7f2ad37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:30.001 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:29.998+0000 7f2ad37fe640 10 monclient(hunting): my global_id is 4196 2024-03-25T18:16:30.001 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:29.998+0000 7f2ad37fe640 1 -- 172.21.15.97:0/1633116660 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ad8004eb0 con 0x7f2aec0b10d0 2024-03-25T18:16:30.001 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:29.998+0000 7f2ad37fe640 1 -- 172.21.15.97:0/1633116660 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1237278836 0 0) 0x7f2aec1d80c0 con 0x7f2aec0b4360 2024-03-25T18:16:30.001 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:29.998+0000 7f2ad37fe640 10 monclient(hunting): _negotiate 2024-03-25T18:16:30.001 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:29.998+0000 7f2ad37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:30.001 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:29.998+0000 7f2ad37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:30.002 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:29.998+0000 7f2ad37fe640 10 monclient(hunting): my global_id is 4201 2024-03-25T18:16:30.002 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:29.998+0000 7f2ad37fe640 1 -- 172.21.15.97:0/1633116660 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2aec1d7490 con 0x7f2aec0b4360 2024-03-25T18:16:30.002 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:29.998+0000 7f2ad37fe640 1 -- 172.21.15.97:0/1633116660 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3213918175 0 0) 0x7f2aec0ac140 con 0x7f2aec0a9470 2024-03-25T18:16:30.002 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:29.998+0000 7f2ad37fe640 10 monclient(hunting): _negotiate 2024-03-25T18:16:30.002 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:29.998+0000 7f2ad37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:30.002 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:29.998+0000 7f2ad37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:30.002 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:29.998+0000 7f2ad37fe640 10 monclient(hunting): my global_id is 4203 2024-03-25T18:16:30.002 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:29.998+0000 7f2ad37fe640 1 -- 172.21.15.97:0/1633116660 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2aec1d80c0 con 0x7f2aec0a9470 2024-03-25T18:16:30.003 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:30.002+0000 7f2ad37fe640 1 -- 172.21.15.97:0/1633116660 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1476310046 0 0) 0x7f2aec1d7490 con 0x7f2aec0b4360 2024-03-25T18:16:30.003 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:30.002+0000 7f2ad37fe640 1 -- 172.21.15.97:0/1633116660 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2aec0ac140 con 0x7f2aec0b4360 2024-03-25T18:16:30.003 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:30.002+0000 7f2ad37fe640 1 -- 172.21.15.97:0/1633116660 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f2adc001830 con 0x7f2aec0b4360 2024-03-25T18:16:30.003 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:30.002+0000 7f2ad37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:30.004 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:30.002+0000 7f2ad37fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:16:30.004 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:30.002+0000 7f2ad37fe640 10 monclient(hunting): dump: 2024-03-25T18:16:30.004 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:16:30.004 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:16:30.004 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:30.004 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:30.004 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:30.004 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:16:30.004 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:16:30.004 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:16:30.004 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:16:30.004 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:16:30.004 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:30.002+0000 7f2ad37fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:16:30.005 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:30.003+0000 7f2ad37fe640 1 -- 172.21.15.97:0/1633116660 <== mon.2 v1:172.21.15.97:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1234556946 0 0) 0x7f2ad8004eb0 con 0x7f2aec0b10d0 2024-03-25T18:16:30.005 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:30.003+0000 7f2ad37fe640 1 -- 172.21.15.97:0/1633116660 --> v1:172.21.15.97:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2aec1d7490 con 0x7f2aec0b10d0 2024-03-25T18:16:30.006 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:30.003+0000 7f2ad37fe640 1 -- 172.21.15.97:0/1633116660 <== mon.2 v1:172.21.15.97:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f2ae0001cb0 con 0x7f2aec0b10d0 2024-03-25T18:16:30.006 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:30.003+0000 7f2ad37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:30.006 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:30.003+0000 7f2ad37fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:16:30.006 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:30.003+0000 7f2ad37fe640 10 monclient(hunting): dump: 2024-03-25T18:16:30.006 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:16:30.006 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:16:30.006 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:30.006 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:30.006 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:30.006 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:16:30.006 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:16:30.006 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:16:30.006 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:16:30.006 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:16:30.006 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:30.003+0000 7f2ad37fe640 1 -- 172.21.15.97:0/1633116660 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 126265170 0 0) 0x7f2aec1d80c0 con 0x7f2aec0a9470 2024-03-25T18:16:30.007 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:30.003+0000 7f2ad37fe640 1 -- 172.21.15.97:0/1633116660 --> v1:172.21.15.97:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2ad8004eb0 con 0x7f2aec0a9470 2024-03-25T18:16:30.007 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:30.003+0000 7f2ad37fe640 1 -- 172.21.15.97:0/1633116660 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f2ae4001a20 con 0x7f2aec0a9470 2024-03-25T18:16:30.007 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:30.003+0000 7f2ad37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:30.007 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:30.003+0000 7f2ad37fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:16:30.007 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:30.003+0000 7f2ad37fe640 10 monclient(hunting): dump: 2024-03-25T18:16:30.007 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:16:30.007 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:16:30.007 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:30.007 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:30.007 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:30.007 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:16:30.007 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:16:30.007 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:16:30.007 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:16:30.007 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:16:30.009 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:30.006+0000 7f2ad37fe640 1 -- 172.21.15.97:0/1633116660 <== mon.1 v1:172.21.15.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 577222654 0 0) 0x7f2aec0ac140 con 0x7f2aec0b4360 2024-03-25T18:16:30.009 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:30.006+0000 7f2ad37fe640 1 -- 172.21.15.97:0/1633116660 >> v1:172.21.15.97:6790/0 conn(0x7f2aec0b10d0 legacy=0x7f2aec13e9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:30.009 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:30.006+0000 7f2ad37fe640 1 -- 172.21.15.97:0/1633116660 >> v1:172.21.15.97:6789/0 conn(0x7f2aec0a9470 legacy=0x7f2aec1d79b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:30.009 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:30.006+0000 7f2ad37fe640 10 monclient: _finish_hunting 0 2024-03-25T18:16:30.009 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:30.006+0000 7f2ad37fe640 1 monclient: found mon.b 2024-03-25T18:16:30.009 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:30.006+0000 7f2ad37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:30.009 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:30.006+0000 7f2ad37fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:16:30.009 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:30.006+0000 7f2ad37fe640 1 -- 172.21.15.97:0/1633116660 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2aec13f100 con 0x7f2aec0b4360 2024-03-25T18:16:30.009 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:30.006+0000 7f2ad37fe640 10 monclient: _finish_auth 0 2024-03-25T18:16:30.010 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:30.006+0000 7f2ad37fe640 10 monclient: _check_auth_tickets 2024-03-25T18:16:30.010 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:30.006+0000 7f2ad37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:30.010 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:30.006+0000 7f2af1748640 5 monclient: authenticate success, global_id 4201 2024-03-25T18:16:30.010 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:30.006+0000 7f2af1748640 10 monclient: _renew_subs 2024-03-25T18:16:30.010 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:30.006+0000 7f2af1748640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:16:30.010 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:30.006+0000 7f2af1748640 1 -- 172.21.15.97:0/1633116660 --> v1:172.21.15.98:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2aec13dd50 con 0x7f2aec0b4360 2024-03-25T18:16:30.010 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:30.006+0000 7f2af1748640 10 client.4201.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:16:30.010 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:30.006+0000 7f2af1748640 10 monclient: _renew_subs 2024-03-25T18:16:30.010 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:30.006+0000 7f2af1748640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:16:30.010 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:30.006+0000 7f2af1748640 1 -- 172.21.15.97:0/1633116660 --> v1:172.21.15.98:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2aec13b5a0 con 0x7f2aec0b4360 2024-03-25T18:16:30.010 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:30.006+0000 7f2af1748640 1 librados: init done 2024-03-25T18:16:30.010 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:30.007+0000 7f2af1748640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:30.010 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:30.007+0000 7f2af1748640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:30.010 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:30.007+0000 7f2af1748640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:16:30.010 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:30.007+0000 7f2af1748640 1 -- 172.21.15.97:0/1633116660 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2aec1d7620 con 0x7f2aec0b4360 2024-03-25T18:16:30.013 INFO:tasks.ceph.osd.7.smithi098.stderr:==00:00:00:09.492 35255== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:30.025 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:30.024+0000 7f2ad37fe640 1 -- 172.21.15.97:0/1633116660 <== mon.1 v1:172.21.15.98:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2adc001ce0 con 0x7f2aec0b4360 2024-03-25T18:16:30.026 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:30.024+0000 7f2ad37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:30.026 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:30.024+0000 7f2ad37fe640 1 -- 172.21.15.97:0/1633116660 <== mon.1 v1:172.21.15.98:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f2adc0037b0 con 0x7f2aec0b4360 2024-03-25T18:16:30.026 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:30.024+0000 7f2ad37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:30.026 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:30.024+0000 7f2ad37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:16:30.026 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:30.024+0000 7f2ad37fe640 10 monclient: dump: 2024-03-25T18:16:30.026 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:16:30.026 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:16:30.026 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:30.026 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:30.026 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:30.026 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:16:30.026 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:16:30.027 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:16:30.027 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:16:30.027 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:16:30.027 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:30.024+0000 7f2ae97fa640 4 set_mon_vals no callback set 2024-03-25T18:16:30.027 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:30.025+0000 7f2ad37fe640 1 -- 172.21.15.97:0/1633116660 <== mon.1 v1:172.21.15.98:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2377622137 0 0) 0x7f2adc0039d0 con 0x7f2aec0b4360 2024-03-25T18:16:30.027 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:30.025+0000 7f2aeaffd640 10 monclient: get_auth_request con 0x7f2ad806e0b0 auth_method 0 2024-03-25T18:16:30.027 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:30.026+0000 7f2ad37fe640 10 client.4201.objecter ms_handle_connect 0x7f2ad806e0b0 2024-03-25T18:16:30.028 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:30.027+0000 7f2ad37fe640 1 -- 172.21.15.97:0/1633116660 <== mon.1 v1:172.21.15.98:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 430711918 0 0) 0x7f2adc054660 con 0x7f2aec0b4360 2024-03-25T18:16:30.028 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:30.027+0000 7f2ad37fe640 10 client.4201.objecter ms_dispatch 0x7f2aec000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:16:30.028 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:30.027+0000 7f2ad37fe640 3 client.4201.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:16:30.028 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:30.027+0000 7f2ad37fe640 3 client.4201.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:16:30.028 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:30.027+0000 7f2ad37fe640 20 client.4201.objecter dump_active .. 0 homeless 2024-03-25T18:16:30.046 INFO:tasks.ceph.osd.2.smithi097.stderr:==00:00:00:09.547 35247== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:30.148 INFO:tasks.ceph.osd.4.smithi098.stderr:==00:00:00:09.654 35252== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:30.191 INFO:tasks.ceph.osd.5.smithi098.stderr:==00:00:00:09.696 35253== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:30.204 INFO:tasks.ceph.osd.3.smithi097.stderr:==00:00:00:09.712 35246== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:30.207 INFO:tasks.ceph.osd.1.smithi097.stderr:==00:00:00:09.696 35248== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:30.238 INFO:tasks.ceph.osd.6.smithi098.stderr:==00:00:00:09.736 35254== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:30.254 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:30.253+0000 7f2ad37fe640 1 -- 172.21.15.97:0/1633116660 <== mon.1 v1:172.21.15.98:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2adc050170 con 0x7f2aec0b4360 2024-03-25T18:16:30.254 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:30.253+0000 7f2ad37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:30.254 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:30.253+0000 7f2ad37fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:16:30.254 INFO:tasks.ceph.osd.0.smithi097.stderr:==00:00:00:09.762 35245== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:30.278 INFO:tasks.ceph.osd.7.smithi098.stderr:==00:00:00:09.757 35255== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:30.311 INFO:tasks.ceph.osd.2.smithi097.stderr:==00:00:00:09.812 35247== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:30.427 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:30.426+0000 7f2af1748640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:30.427 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:30.426+0000 7f2af1748640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:30.427 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:30.426+0000 7f2af1748640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:16:30.427 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:30.426+0000 7f2af1748640 1 -- 172.21.15.97:0/1633116660 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2aec1d7280 con 0x7f2aec0b4360 2024-03-25T18:16:30.441 INFO:tasks.ceph.osd.4.smithi098.stderr:==00:00:00:09.948 35252== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:30.443 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:30.442+0000 7f2ad37fe640 1 -- 172.21.15.97:0/1633116660 <== mon.1 v1:172.21.15.98:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2169427412) 0x7f2adc0520a0 con 0x7f2aec0b4360 2024-03-25T18:16:30.443 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:30.442+0000 7f2ad37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:30.443 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:30.442+0000 7f2ad37fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:16:30.443 INFO:teuthology.misc.health.smithi097.stdout: 2024-03-25T18:16:30.443 INFO:teuthology.misc.health.smithi097.stdout:{"epoch":10,"fsid":"4c3d472c-79b8-486b-9e3d-2738d4306754","created":"2024-03-25T18:16:14.737309+0000","modified":"2024-03-25T18:16:20.357492+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:20.357492+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cb5152b9-262f-45c3-94b8-18d0e81df558","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6daf8793-0838-473d-a7d6-9a0d2894c75d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"136f1493-3b9c-4f43-a2e7-641482d69dae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"397d538c-4f64-4daf-9e8b-662d09f91327","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4f1df08-663f-4bda-93f8-3878ab7c24b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c4b18d70-a3ce-4b15-9b70-eae5525bd761","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8dd4033a-de0e-4f8a-96dc-59f3c1f825b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0077722e-ffa6-4a3d-9891-929d9869b286","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:16:30.446 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:30.444+0000 7f2af1748640 10 librados: watch_flush enter 2024-03-25T18:16:30.446 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:30.444+0000 7f2af1748640 10 librados: watch_flush exit 2024-03-25T18:16:30.446 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:30.444+0000 7f2af1748640 20 client.4201.objecter shutdown clearing up homeless session... 2024-03-25T18:16:30.446 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:30.444+0000 7f2af1748640 10 client.4201.objecter successfully canceled tick 2024-03-25T18:16:30.446 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:30.444+0000 7f2af1748640 1 -- 172.21.15.97:0/1633116660 >> v1:172.21.15.98:6800/800387573 conn(0x7f2ad806e0b0 legacy=0x7f2ad8070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:30.446 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:30.444+0000 7f2af1748640 10 monclient: shutdown 2024-03-25T18:16:30.446 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:30.444+0000 7f2af1748640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:30.446 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:30.444+0000 7f2af1748640 1 -- 172.21.15.97:0/1633116660 >> v1:172.21.15.98:6789/0 conn(0x7f2aec0b4360 legacy=0x7f2aec1331c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:30.446 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:30.444+0000 7f2af1748640 1 -- 172.21.15.97:0/1633116660 shutdown_connections 2024-03-25T18:16:30.446 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:30.444+0000 7f2af1748640 1 -- 172.21.15.97:0/1633116660 >> 172.21.15.97:0/1633116660 conn(0x7f2aec0617a0 msgr2=0x7f2aec0b6280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:30.446 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:30.444+0000 7f2af1748640 1 -- 172.21.15.97:0/1633116660 shutdown_connections 2024-03-25T18:16:30.446 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:30.444+0000 7f2af1748640 1 -- 172.21.15.97:0/1633116660 wait complete. 2024-03-25T18:16:30.446 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:30.444+0000 7f2af1748640 1 librados: shutdown 2024-03-25T18:16:30.454 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:16:30.466 INFO:tasks.ceph.osd.5.smithi098.stderr:==00:00:00:09.971 35253== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:30.479 INFO:tasks.ceph.osd.3.smithi097.stderr:==00:00:00:09.987 35246== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:30.502 INFO:tasks.ceph.osd.6.smithi098.stderr:==00:00:00:10.000 35254== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:30.505 INFO:tasks.ceph.osd.1.smithi097.stderr:==00:00:00:09.994 35248== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:30.530 INFO:tasks.ceph.osd.0.smithi097.stderr:==00:00:00:10.038 35245== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:30.549 INFO:tasks.ceph.osd.7.smithi098.stderr:==00:00:00:10.028 35255== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:30.575 INFO:tasks.ceph.osd.2.smithi097.stderr:==00:00:00:10.076 35247== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:30.736 INFO:tasks.ceph.osd.5.smithi098.stderr:==00:00:00:10.241 35253== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:30.741 INFO:tasks.ceph.osd.4.smithi098.stderr:==00:00:00:10.248 35252== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:30.754 INFO:tasks.ceph.osd.3.smithi097.stderr:==00:00:00:10.262 35246== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:30.771 INFO:tasks.ceph.osd.6.smithi098.stderr:==00:00:00:10.269 35254== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:30.797 INFO:tasks.ceph.osd.1.smithi097.stderr:==00:00:00:10.286 35248== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:30.800 INFO:tasks.ceph.osd.0.smithi097.stderr:==00:00:00:10.308 35245== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:30.813 INFO:tasks.ceph.osd.7.smithi098.stderr:==00:00:00:10.292 35255== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:30.845 INFO:tasks.ceph.osd.2.smithi097.stderr:==00:00:00:10.346 35247== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:31.000 INFO:tasks.ceph.osd.5.smithi098.stderr:==00:00:00:10.505 35253== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:31.023 INFO:tasks.ceph.osd.3.smithi097.stderr:==00:00:00:10.530 35246== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:31.036 INFO:tasks.ceph.osd.4.smithi098.stderr:==00:00:00:10.543 35252== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:31.095 INFO:tasks.ceph.osd.1.smithi097.stderr:==00:00:00:10.584 35248== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:31.114 INFO:tasks.ceph.osd.2.smithi097.stderr:==00:00:00:10.615 35247== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:31.253 INFO:tasks.ceph.osd.6.smithi098.stderr:==00:00:00:10.751 35254== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:31.274 INFO:tasks.ceph.osd.0.smithi097.stderr:==00:00:00:10.782 35245== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:31.280 INFO:tasks.ceph.osd.7.smithi098.stderr:==00:00:00:10.758 35255== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:31.290 INFO:tasks.ceph.osd.3.smithi097.stderr:==00:00:00:10.798 35246== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:31.308 INFO:tasks.ceph.osd.6.smithi098.stderr:==00:00:00:10.805 35254== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:31.329 INFO:tasks.ceph.osd.0.smithi097.stderr:==00:00:00:10.837 35245== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:31.334 INFO:tasks.ceph.osd.7.smithi098.stderr:==00:00:00:10.813 35255== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:31.388 INFO:tasks.ceph.osd.1.smithi097.stderr:==00:00:00:10.877 35248== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:31.490 INFO:tasks.ceph.osd.5.smithi098.stderr:==00:00:00:10.995 35253== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:31.544 INFO:tasks.ceph.osd.5.smithi098.stderr:==00:00:00:11.049 35253== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:31.566 INFO:tasks.ceph.osd.3.smithi097.stderr:==00:00:00:11.074 35246== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:31.585 INFO:tasks.ceph.osd.2.smithi097.stderr:==00:00:00:11.086 35247== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:31.638 INFO:tasks.ceph.osd.2.smithi097.stderr:==00:00:00:11.139 35247== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:31.645 INFO:tasks.ceph.osd.4.smithi098.stderr:==00:00:00:11.151 35252== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:31.677 INFO:tasks.ceph.osd.1.smithi097.stderr:==00:00:00:11.166 35248== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:31.700 INFO:tasks.ceph.osd.4.smithi098.stderr:==00:00:00:11.206 35252== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:31.832 INFO:tasks.ceph.osd.3.smithi097.stderr:==00:00:00:11.340 35246== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:31.966 INFO:tasks.ceph.osd.1.smithi097.stderr:==00:00:00:11.455 35248== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:32.347 INFO:tasks.ceph.osd.3.smithi097.stderr:==00:00:00:11.855 35246== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:32.401 INFO:tasks.ceph.osd.3.smithi097.stderr:==00:00:00:11.909 35246== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:32.528 INFO:tasks.ceph.osd.1.smithi097.stderr:==00:00:00:12.017 35248== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:32.584 INFO:tasks.ceph.osd.1.smithi097.stderr:==00:00:00:12.073 35248== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:36.559 INFO:tasks.ceph.osd.6.smithi098.stderr:==00:00:00:16.057 35254== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:36.601 INFO:tasks.ceph.osd.7.smithi098.stderr:==00:00:00:16.079 35255== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:36.706 INFO:tasks.ceph.osd.5.smithi098.stderr:==00:00:00:16.211 35253== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:36.888 INFO:tasks.ceph.osd.4.smithi098.stderr:==00:00:00:16.394 35252== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:37.262 DEBUG:teuthology.orchestra.run.smithi097:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:16:37.395 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.394+0000 7ffad7715640 10 monclient: get_monmap_and_config 2024-03-25T18:16:37.395 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.394+0000 7ffad7715640 10 monclient: build_initial_monmap 2024-03-25T18:16:37.395 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.394+0000 7ffad7715640 10 monclient: monmap: 2024-03-25T18:16:37.395 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:16:37.395 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:16:37.396 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:16:37.395657+0000 2024-03-25T18:16:37.396 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:16:37.395657+0000 2024-03-25T18:16:37.396 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:37.396 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:16:37.396 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:16:37.396 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T18:16:37.396 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-b 2024-03-25T18:16:37.396 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:16:37.397 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.395+0000 7ffad7715640 1 Processor -- start 2024-03-25T18:16:37.397 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.395+0000 7ffad7715640 1 -- start start 2024-03-25T18:16:37.398 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.395+0000 7ffad7715640 10 monclient: init 2024-03-25T18:16:37.398 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.395+0000 7ffad7715640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:37.398 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.395+0000 7ffad7715640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:16:37.398 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.395+0000 7ffad7715640 10 monclient(hunting): picked mon.noname-a con 0x7ffad00b4d60 addr v1:172.21.15.97:6789/0 2024-03-25T18:16:37.398 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.395+0000 7ffad7715640 10 monclient(hunting): picked mon.noname-c con 0x7ffad00b1730 addr v1:172.21.15.98:6789/0 2024-03-25T18:16:37.398 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.395+0000 7ffad7715640 10 monclient(hunting): picked mon.noname-b con 0x7ffad014be90 addr v1:172.21.15.97:6790/0 2024-03-25T18:16:37.398 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.395+0000 7ffad7715640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffad00af940 con 0x7ffad00b4d60 2024-03-25T18:16:37.398 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.395+0000 7ffad7715640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffad00b0a70 con 0x7ffad00b1730 2024-03-25T18:16:37.398 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.395+0000 7ffad7715640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffad00b0d50 con 0x7ffad014be90 2024-03-25T18:16:37.398 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.395+0000 7ffad7715640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:37.398 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.395+0000 7ffad7715640 10 monclient(hunting): authenticate will time out at 7609.671803s 2024-03-25T18:16:37.398 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.396+0000 7ffad5cb2640 1 --1- >> v1:172.21.15.97:6790/0 conn(0x7ffad014be90 0x7ffad014e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6790/0 says I am v1:172.21.15.97:50494/0 (socket says 172.21.15.97:50494) 2024-03-25T18:16:37.399 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.396+0000 7ffad5cb2640 1 -- 172.21.15.97:0/2782930169 learned_addr learned my addr 172.21.15.97:0/2782930169 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:16:37.400 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.399+0000 7ffac7fff640 1 -- 172.21.15.97:0/2782930169 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3192192873 0 0) 0x7ffad00b0a70 con 0x7ffad00b1730 2024-03-25T18:16:37.400 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.399+0000 7ffac7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:16:37.400 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.399+0000 7ffac7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:37.400 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.399+0000 7ffac7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:37.400 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.399+0000 7ffac7fff640 10 monclient(hunting): my global_id is 4204 2024-03-25T18:16:37.401 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.399+0000 7ffac7fff640 1 -- 172.21.15.97:0/2782930169 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffabc003520 con 0x7ffad00b1730 2024-03-25T18:16:37.402 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.400+0000 7ffac7fff640 1 -- 172.21.15.97:0/2782930169 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 163274609 0 0) 0x7ffad00af940 con 0x7ffad00b4d60 2024-03-25T18:16:37.402 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.400+0000 7ffac7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:16:37.402 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.400+0000 7ffac7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:37.402 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.400+0000 7ffac7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:37.402 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.400+0000 7ffac7fff640 10 monclient(hunting): my global_id is 4206 2024-03-25T18:16:37.402 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.400+0000 7ffac7fff640 1 -- 172.21.15.97:0/2782930169 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffad00b0a70 con 0x7ffad00b4d60 2024-03-25T18:16:37.402 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.400+0000 7ffac7fff640 1 -- 172.21.15.97:0/2782930169 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1172146331 0 0) 0x7ffad00b0d50 con 0x7ffad014be90 2024-03-25T18:16:37.403 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.400+0000 7ffac7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:16:37.403 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.400+0000 7ffac7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:37.403 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.400+0000 7ffac7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:37.403 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.400+0000 7ffac7fff640 10 monclient(hunting): my global_id is 4199 2024-03-25T18:16:37.403 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.400+0000 7ffac7fff640 1 -- 172.21.15.97:0/2782930169 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffad00af940 con 0x7ffad014be90 2024-03-25T18:16:37.406 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.404+0000 7ffac7fff640 1 -- 172.21.15.97:0/2782930169 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3689490943 0 0) 0x7ffad00b0a70 con 0x7ffad00b4d60 2024-03-25T18:16:37.406 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.404+0000 7ffac7fff640 1 -- 172.21.15.97:0/2782930169 >> v1:172.21.15.97:6790/0 conn(0x7ffad014be90 legacy=0x7ffad014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:37.406 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.404+0000 7ffac7fff640 1 -- 172.21.15.97:0/2782930169 >> v1:172.21.15.98:6789/0 conn(0x7ffad00b1730 legacy=0x7ffad00b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:37.406 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.404+0000 7ffac7fff640 10 monclient: _finish_hunting 0 2024-03-25T18:16:37.406 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.404+0000 7ffac7fff640 1 monclient: found mon.noname-a 2024-03-25T18:16:37.406 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.404+0000 7ffac7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:37.406 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.404+0000 7ffac7fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.97:6789/0 2024-03-25T18:16:37.406 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.404+0000 7ffac7fff640 1 -- 172.21.15.97:0/2782930169 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffad00b1030 con 0x7ffad00b4d60 2024-03-25T18:16:37.406 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.404+0000 7ffac7fff640 10 monclient: _finish_auth 0 2024-03-25T18:16:37.406 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.404+0000 7ffac7fff640 10 monclient: _check_auth_tickets 2024-03-25T18:16:37.407 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.404+0000 7ffac7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:37.407 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.404+0000 7ffac7fff640 1 -- 172.21.15.97:0/2782930169 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7ffacc001670 con 0x7ffad00b4d60 2024-03-25T18:16:37.407 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.404+0000 7ffac7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:37.407 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.405+0000 7ffac7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:16:37.407 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.405+0000 7ffac7fff640 10 monclient: dump: 2024-03-25T18:16:37.407 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:16:37.407 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:16:37.407 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:37.407 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:37.407 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:37.407 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:16:37.407 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:16:37.407 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:16:37.407 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:16:37.407 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:16:37.407 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.405+0000 7ffad7715640 5 monclient: authenticate success, global_id 4206 2024-03-25T18:16:37.408 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.405+0000 7ffad7715640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:37.408 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.407+0000 7ffac7fff640 1 -- 172.21.15.97:0/2782930169 <== mon.0 v1:172.21.15.97:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffacc0017d0 con 0x7ffad00b4d60 2024-03-25T18:16:37.409 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.407+0000 7ffac7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:37.409 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.407+0000 7ffac7fff640 1 -- 172.21.15.97:0/2782930169 <== mon.0 v1:172.21.15.97:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7ffacc001be0 con 0x7ffad00b4d60 2024-03-25T18:16:37.409 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.407+0000 7ffac7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:37.409 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.407+0000 7ffac7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:16:37.409 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.407+0000 7ffac7fff640 10 monclient: dump: 2024-03-25T18:16:37.409 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:16:37.409 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:16:37.409 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:37.409 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:37.409 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:37.409 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:16:37.409 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:16:37.409 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:16:37.409 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:16:37.409 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:16:37.410 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.407+0000 7ffad7715640 10 monclient: get_monmap_and_config success 2024-03-25T18:16:37.410 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.407+0000 7ffad7715640 4 set_mon_vals no callback set 2024-03-25T18:16:37.410 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.408+0000 7ffad7715640 10 monclient: shutdown 2024-03-25T18:16:37.411 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.408+0000 7ffad7715640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:37.411 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.408+0000 7ffad7715640 1 -- 172.21.15.97:0/2782930169 >> v1:172.21.15.97:6789/0 conn(0x7ffad00b4d60 legacy=0x7ffad013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:37.411 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.408+0000 7ffad7715640 1 -- 172.21.15.97:0/2782930169 shutdown_connections 2024-03-25T18:16:37.411 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.408+0000 7ffad7715640 1 -- 172.21.15.97:0/2782930169 >> 172.21.15.97:0/2782930169 conn(0x7ffad00617a0 msgr2=0x7ffad0061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:37.411 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.409+0000 7ffad7715640 1 -- 172.21.15.97:0/2782930169 shutdown_connections 2024-03-25T18:16:37.411 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.409+0000 7ffad7715640 1 -- 172.21.15.97:0/2782930169 wait complete. 2024-03-25T18:16:37.411 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.410+0000 7ffad7715640 10 monclient: build_initial_monmap 2024-03-25T18:16:37.412 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.410+0000 7ffad7715640 10 monclient: monmap: 2024-03-25T18:16:37.412 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:16:37.412 INFO:teuthology.misc.health.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:16:37.412 INFO:teuthology.misc.health.smithi097.stderr:last_changed 0.000000 2024-03-25T18:16:37.412 INFO:teuthology.misc.health.smithi097.stderr:created 0.000000 2024-03-25T18:16:37.412 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:37.412 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 1 2024-03-25T18:16:37.412 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:16:37.412 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T18:16:37.412 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-c 2024-03-25T18:16:37.412 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:16:37.413 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.411+0000 7ffad7715640 1 librados: starting msgr at 2024-03-25T18:16:37.413 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.411+0000 7ffad7715640 1 librados: starting objecter 2024-03-25T18:16:37.414 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.411+0000 7ffad7715640 1 Processor -- start 2024-03-25T18:16:37.414 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.411+0000 7ffad7715640 1 -- start start 2024-03-25T18:16:37.414 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.411+0000 7ffad7715640 1 librados: setting wanted keys 2024-03-25T18:16:37.414 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.411+0000 7ffad7715640 1 librados: calling monclient init 2024-03-25T18:16:37.414 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.411+0000 7ffad7715640 10 monclient: init 2024-03-25T18:16:37.414 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.412+0000 7ffad7715640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:37.414 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.412+0000 7ffad7715640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:16:37.414 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.412+0000 7ffad7715640 10 monclient(hunting): picked mon.noname-a con 0x7ffad00b1730 addr v1:172.21.15.97:6789/0 2024-03-25T18:16:37.414 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.412+0000 7ffad7715640 10 monclient(hunting): picked mon.noname-b con 0x7ffad00b4d60 addr v1:172.21.15.98:6789/0 2024-03-25T18:16:37.414 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.412+0000 7ffad4cb0640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7ffad00b4d60 0x7ffad01365e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.97:54158/0 (socket says 172.21.15.97:54158) 2024-03-25T18:16:37.415 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.413+0000 7ffad4cb0640 1 -- 172.21.15.97:0/891310432 learned_addr learned my addr 172.21.15.97:0/891310432 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:16:37.415 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.413+0000 7ffad7715640 10 monclient(hunting): picked mon.noname-c con 0x7ffad014be90 addr v1:172.21.15.97:6790/0 2024-03-25T18:16:37.415 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.413+0000 7ffad7715640 1 -- 172.21.15.97:0/891310432 --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffad0143430 con 0x7ffad00b1730 2024-03-25T18:16:37.415 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.413+0000 7ffad7715640 1 -- 172.21.15.97:0/891310432 --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffad01dbbb0 con 0x7ffad00b4d60 2024-03-25T18:16:37.415 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.413+0000 7ffad7715640 1 -- 172.21.15.97:0/891310432 --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffad01dc7e0 con 0x7ffad014be90 2024-03-25T18:16:37.415 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.413+0000 7ffad7715640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:37.415 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.413+0000 7ffad7715640 10 monclient(hunting): authenticate will time out at 7609.689184s 2024-03-25T18:16:37.416 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.413+0000 7ffac5ffb640 10 client.?.objecter ms_handle_connect 0x7ffad014be90 2024-03-25T18:16:37.416 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.413+0000 7ffac5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:37.416 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.414+0000 7ffac5ffb640 10 client.?.objecter ms_handle_connect 0x7ffad00b1730 2024-03-25T18:16:37.416 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.414+0000 7ffac5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:37.416 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.414+0000 7ffac5ffb640 10 client.?.objecter ms_handle_connect 0x7ffad00b4d60 2024-03-25T18:16:37.416 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.414+0000 7ffac5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:37.416 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.415+0000 7ffac5ffb640 1 -- 172.21.15.97:0/891310432 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2101599984 0 0) 0x7ffad01dc7e0 con 0x7ffad014be90 2024-03-25T18:16:37.416 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.415+0000 7ffac5ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:16:37.416 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.415+0000 7ffac5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:37.416 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.415+0000 7ffac5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:37.417 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.415+0000 7ffac5ffb640 10 monclient(hunting): my global_id is 4202 2024-03-25T18:16:37.417 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.415+0000 7ffac5ffb640 1 -- 172.21.15.97:0/891310432 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffab4004eb0 con 0x7ffad014be90 2024-03-25T18:16:37.417 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.416+0000 7ffac5ffb640 1 -- 172.21.15.97:0/891310432 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2840447919 0 0) 0x7ffad01dbbb0 con 0x7ffad00b4d60 2024-03-25T18:16:37.417 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.416+0000 7ffac5ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:16:37.417 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.416+0000 7ffac5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:37.417 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.416+0000 7ffac5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:37.418 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.416+0000 7ffac5ffb640 10 monclient(hunting): my global_id is 4207 2024-03-25T18:16:37.418 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.416+0000 7ffac5ffb640 1 -- 172.21.15.97:0/891310432 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffad01dc7e0 con 0x7ffad00b4d60 2024-03-25T18:16:37.419 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.418+0000 7ffac5ffb640 1 -- 172.21.15.97:0/891310432 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 253184132 0 0) 0x7ffad0143430 con 0x7ffad00b1730 2024-03-25T18:16:37.419 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.418+0000 7ffac5ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:16:37.419 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.418+0000 7ffac5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:37.419 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.418+0000 7ffac5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:37.419 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.418+0000 7ffac5ffb640 10 monclient(hunting): my global_id is 4209 2024-03-25T18:16:37.419 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.418+0000 7ffac5ffb640 1 -- 172.21.15.97:0/891310432 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffad01dbbb0 con 0x7ffad00b1730 2024-03-25T18:16:37.421 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.420+0000 7ffac5ffb640 1 -- 172.21.15.97:0/891310432 <== mon.2 v1:172.21.15.97:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1873016554 0 0) 0x7ffab4004eb0 con 0x7ffad014be90 2024-03-25T18:16:37.421 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.420+0000 7ffac5ffb640 1 -- 172.21.15.97:0/891310432 --> v1:172.21.15.97:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffad0143430 con 0x7ffad014be90 2024-03-25T18:16:37.421 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.420+0000 7ffac5ffb640 1 -- 172.21.15.97:0/891310432 <== mon.2 v1:172.21.15.97:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7ffac8001bd0 con 0x7ffad014be90 2024-03-25T18:16:37.421 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.420+0000 7ffac5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:37.421 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.420+0000 7ffac5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:16:37.421 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.420+0000 7ffac5ffb640 10 monclient(hunting): dump: 2024-03-25T18:16:37.421 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:16:37.422 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:16:37.422 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:37.422 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:37.422 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:37.422 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:16:37.422 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:16:37.422 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:16:37.422 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:16:37.422 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:16:37.422 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.420+0000 7ffac5ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:16:37.423 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.422+0000 7ffac5ffb640 1 -- 172.21.15.97:0/891310432 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1063677670 0 0) 0x7ffad01dc7e0 con 0x7ffad00b4d60 2024-03-25T18:16:37.423 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.422+0000 7ffac5ffb640 1 -- 172.21.15.97:0/891310432 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffab4004eb0 con 0x7ffad00b4d60 2024-03-25T18:16:37.423 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.422+0000 7ffac5ffb640 1 -- 172.21.15.97:0/891310432 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7ffac0001950 con 0x7ffad00b4d60 2024-03-25T18:16:37.424 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.422+0000 7ffac5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:37.424 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.422+0000 7ffac5ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:16:37.424 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.422+0000 7ffac5ffb640 10 monclient(hunting): dump: 2024-03-25T18:16:37.424 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:16:37.424 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:16:37.424 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:37.424 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:37.424 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:37.424 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:16:37.424 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:16:37.424 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:16:37.424 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:16:37.424 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:16:37.426 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.423+0000 7ffac5ffb640 1 -- 172.21.15.97:0/891310432 <== mon.2 v1:172.21.15.97:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3361100729 0 0) 0x7ffad0143430 con 0x7ffad014be90 2024-03-25T18:16:37.426 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.423+0000 7ffac5ffb640 1 -- 172.21.15.97:0/891310432 >> v1:172.21.15.98:6789/0 conn(0x7ffad00b4d60 legacy=0x7ffad01365e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:37.427 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.423+0000 7ffac5ffb640 1 -- 172.21.15.97:0/891310432 >> v1:172.21.15.97:6789/0 conn(0x7ffad00b1730 legacy=0x7ffad01dc0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:37.427 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.423+0000 7ffac5ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:16:37.427 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.423+0000 7ffac5ffb640 1 monclient: found mon.c 2024-03-25T18:16:37.427 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.423+0000 7ffac5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:37.427 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.423+0000 7ffac5ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.97:6790/0 2024-03-25T18:16:37.427 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.423+0000 7ffac5ffb640 1 -- 172.21.15.97:0/891310432 --> v1:172.21.15.97:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffad01dc970 con 0x7ffad014be90 2024-03-25T18:16:37.427 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.423+0000 7ffac5ffb640 10 monclient: _finish_auth 0 2024-03-25T18:16:37.427 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.423+0000 7ffac5ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:16:37.427 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.423+0000 7ffac5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:37.427 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.424+0000 7ffad7715640 5 monclient: authenticate success, global_id 4202 2024-03-25T18:16:37.427 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.424+0000 7ffad7715640 10 monclient: _renew_subs 2024-03-25T18:16:37.427 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.424+0000 7ffad7715640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.97:6790/0 2024-03-25T18:16:37.427 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.424+0000 7ffad7715640 1 -- 172.21.15.97:0/891310432 --> v1:172.21.15.97:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffad013a370 con 0x7ffad014be90 2024-03-25T18:16:37.428 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.424+0000 7ffad7715640 10 client.4202.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:16:37.428 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.424+0000 7ffad7715640 10 monclient: _renew_subs 2024-03-25T18:16:37.428 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.424+0000 7ffad7715640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.97:6790/0 2024-03-25T18:16:37.428 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.424+0000 7ffad7715640 1 -- 172.21.15.97:0/891310432 --> v1:172.21.15.97:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ffad0136ed0 con 0x7ffad014be90 2024-03-25T18:16:37.428 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.424+0000 7ffad7715640 1 librados: init done 2024-03-25T18:16:37.428 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.425+0000 7ffad7715640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:37.428 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.425+0000 7ffad7715640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:37.428 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.425+0000 7ffad7715640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.97:6790/0 2024-03-25T18:16:37.428 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.425+0000 7ffad7715640 1 -- 172.21.15.97:0/891310432 --> v1:172.21.15.97:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffad00afbd0 con 0x7ffad014be90 2024-03-25T18:16:37.450 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.448+0000 7ffac5ffb640 1 -- 172.21.15.97:0/891310432 <== mon.2 v1:172.21.15.97:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffac8001fe0 con 0x7ffad014be90 2024-03-25T18:16:37.451 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.448+0000 7ffac5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:37.451 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.448+0000 7ffac5ffb640 1 -- 172.21.15.97:0/891310432 <== mon.2 v1:172.21.15.97:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7ffac8004d00 con 0x7ffad014be90 2024-03-25T18:16:37.451 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.448+0000 7ffac5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:37.451 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.448+0000 7ffac5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:16:37.451 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.448+0000 7ffac5ffb640 10 monclient: dump: 2024-03-25T18:16:37.451 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:16:37.451 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:16:37.451 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:37.451 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:37.451 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:37.451 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:16:37.451 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:16:37.451 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:16:37.452 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:16:37.452 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:16:37.452 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.448+0000 7ffac5ffb640 1 -- 172.21.15.97:0/891310432 <== mon.2 v1:172.21.15.97:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2377622137 0 0) 0x7ffac8004f20 con 0x7ffad014be90 2024-03-25T18:16:37.452 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.448+0000 7ffac77fe640 4 set_mon_vals no callback set 2024-03-25T18:16:37.452 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.449+0000 7ffad54b1640 10 monclient: get_auth_request con 0x7ffab406dfb0 auth_method 0 2024-03-25T18:16:37.452 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.449+0000 7ffac5ffb640 10 client.4202.objecter ms_handle_connect 0x7ffab406dfb0 2024-03-25T18:16:37.454 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.453+0000 7ffac5ffb640 1 -- 172.21.15.97:0/891310432 <== mon.2 v1:172.21.15.97:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 430711918 0 0) 0x7ffac8054a40 con 0x7ffad014be90 2024-03-25T18:16:37.454 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.453+0000 7ffac5ffb640 10 client.4202.objecter ms_dispatch 0x7ffad0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:16:37.454 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.453+0000 7ffac5ffb640 3 client.4202.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:16:37.454 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.453+0000 7ffac5ffb640 3 client.4202.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:16:37.454 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.453+0000 7ffac5ffb640 20 client.4202.objecter dump_active .. 0 homeless 2024-03-25T18:16:37.491 INFO:tasks.ceph.osd.0.smithi097.stderr:==00:00:00:16.999 35245== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:37.553 INFO:tasks.ceph.osd.2.smithi097.stderr:==00:00:00:17.054 35247== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:37.725 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.724+0000 7ffac5ffb640 1 -- 172.21.15.97:0/891310432 <== mon.2 v1:172.21.15.97:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ffac8050550 con 0x7ffad014be90 2024-03-25T18:16:37.726 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.724+0000 7ffac5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:37.726 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.724+0000 7ffac5ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:16:37.971 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.969+0000 7ffad7715640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:37.971 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.969+0000 7ffad7715640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:37.971 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.969+0000 7ffad7715640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.97:6790/0 2024-03-25T18:16:37.971 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:37.969+0000 7ffad7715640 1 -- 172.21.15.97:0/891310432 --> v1:172.21.15.97:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffad00627b0 con 0x7ffad014be90 2024-03-25T18:16:37.998 INFO:tasks.ceph.osd.3.smithi097.stderr:==00:00:00:17.506 35246== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:38.018 INFO:teuthology.misc.health.smithi097.stdout: 2024-03-25T18:16:38.018 INFO:teuthology.misc.health.smithi097.stdout:{"epoch":10,"fsid":"4c3d472c-79b8-486b-9e3d-2738d4306754","created":"2024-03-25T18:16:14.737309+0000","modified":"2024-03-25T18:16:20.357492+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:20.357492+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cb5152b9-262f-45c3-94b8-18d0e81df558","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6daf8793-0838-473d-a7d6-9a0d2894c75d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"136f1493-3b9c-4f43-a2e7-641482d69dae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"397d538c-4f64-4daf-9e8b-662d09f91327","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4f1df08-663f-4bda-93f8-3878ab7c24b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c4b18d70-a3ce-4b15-9b70-eae5525bd761","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8dd4033a-de0e-4f8a-96dc-59f3c1f825b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0077722e-ffa6-4a3d-9891-929d9869b286","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:16:38.018 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:38.017+0000 7ffac5ffb640 1 -- 172.21.15.97:0/891310432 <== mon.2 v1:172.21.15.97:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2169427412) 0x7ffac8052480 con 0x7ffad014be90 2024-03-25T18:16:38.018 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:38.017+0000 7ffac5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:38.019 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:38.017+0000 7ffac5ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:16:38.021 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:38.019+0000 7ffad7715640 10 librados: watch_flush enter 2024-03-25T18:16:38.021 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:38.019+0000 7ffad7715640 10 librados: watch_flush exit 2024-03-25T18:16:38.021 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:38.019+0000 7ffad7715640 20 client.4202.objecter shutdown clearing up homeless session... 2024-03-25T18:16:38.021 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:38.019+0000 7ffad7715640 10 client.4202.objecter successfully canceled tick 2024-03-25T18:16:38.021 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:38.019+0000 7ffad7715640 1 -- 172.21.15.97:0/891310432 >> v1:172.21.15.98:6800/800387573 conn(0x7ffab406dfb0 legacy=0x7ffab4070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:38.021 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:38.019+0000 7ffad7715640 10 monclient: shutdown 2024-03-25T18:16:38.021 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:38.019+0000 7ffad7715640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:38.021 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:38.019+0000 7ffad7715640 1 -- 172.21.15.97:0/891310432 >> v1:172.21.15.97:6790/0 conn(0x7ffad014be90 legacy=0x7ffad0141a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:38.022 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:38.020+0000 7ffad7715640 1 -- 172.21.15.97:0/891310432 shutdown_connections 2024-03-25T18:16:38.022 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:38.021+0000 7ffad7715640 1 -- 172.21.15.97:0/891310432 >> 172.21.15.97:0/891310432 conn(0x7ffad00617a0 msgr2=0x7ffad013c0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:38.022 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:38.021+0000 7ffad7715640 1 -- 172.21.15.97:0/891310432 shutdown_connections 2024-03-25T18:16:38.022 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:38.021+0000 7ffad7715640 1 -- 172.21.15.97:0/891310432 wait complete. 2024-03-25T18:16:38.022 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:38.021+0000 7ffad7715640 1 librados: shutdown 2024-03-25T18:16:38.035 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:16:38.552 INFO:tasks.ceph.osd.1.smithi097.stderr:==00:00:00:18.041 35248== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:41.840 INFO:tasks.ceph.osd.7.smithi098.stderr:==00:00:00:21.318 35255== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:41.843 INFO:tasks.ceph.osd.7.smithi098.stderr:==00:00:00:21.322 35255== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:41.978 INFO:tasks.ceph.osd.5.smithi098.stderr:==00:00:00:21.483 35253== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:41.982 INFO:tasks.ceph.osd.5.smithi098.stderr:==00:00:00:21.487 35253== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:42.145 INFO:tasks.ceph.osd.4.smithi098.stderr:==00:00:00:21.651 35252== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:42.149 INFO:tasks.ceph.osd.4.smithi098.stderr:==00:00:00:21.655 35252== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:42.329 INFO:tasks.ceph.osd.6.smithi098.stderr:==00:00:00:21.827 35254== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:42.333 INFO:tasks.ceph.osd.6.smithi098.stderr:==00:00:00:21.831 35254== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:43.161 INFO:tasks.ceph.osd.0.smithi097.stderr:==00:00:00:22.669 35245== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:43.167 INFO:tasks.ceph.osd.0.smithi097.stderr:==00:00:00:22.675 35245== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:43.339 INFO:tasks.ceph.osd.2.smithi097.stderr:==00:00:00:22.840 35247== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:43.344 INFO:tasks.ceph.osd.2.smithi097.stderr:==00:00:00:22.845 35247== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:43.875 INFO:tasks.ceph.osd.3.smithi097.stderr:==00:00:00:23.383 35246== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:43.880 INFO:tasks.ceph.osd.3.smithi097.stderr:==00:00:00:23.388 35246== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:44.453 INFO:tasks.ceph.osd.1.smithi097.stderr:==00:00:00:23.942 35248== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:44.459 INFO:tasks.ceph.osd.1.smithi097.stderr:==00:00:00:23.948 35248== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:44.842 DEBUG:teuthology.orchestra.run.smithi097:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:16:44.939 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.939+0000 7f87d5789640 10 monclient: get_monmap_and_config 2024-03-25T18:16:44.939 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.939+0000 7f87d5789640 10 monclient: build_initial_monmap 2024-03-25T18:16:44.940 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.939+0000 7f87d5789640 10 monclient: monmap: 2024-03-25T18:16:44.940 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:16:44.940 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:16:44.940 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:16:44.939727+0000 2024-03-25T18:16:44.940 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:16:44.939727+0000 2024-03-25T18:16:44.940 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:44.940 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:16:44.940 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:16:44.940 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T18:16:44.940 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-b 2024-03-25T18:16:44.940 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:16:44.942 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.940+0000 7f87d5789640 1 Processor -- start 2024-03-25T18:16:44.942 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.940+0000 7f87d5789640 1 -- start start 2024-03-25T18:16:44.942 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.940+0000 7f87d5789640 10 monclient: init 2024-03-25T18:16:44.942 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.940+0000 7f87d5789640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:44.942 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.940+0000 7f87d5789640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:16:44.942 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.940+0000 7f87d5789640 10 monclient(hunting): picked mon.noname-a con 0x7f87d00b4d60 addr v1:172.21.15.97:6789/0 2024-03-25T18:16:44.942 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.940+0000 7f87d5789640 10 monclient(hunting): picked mon.noname-b con 0x7f87d00b1730 addr v1:172.21.15.97:6790/0 2024-03-25T18:16:44.942 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.940+0000 7f87d5789640 10 monclient(hunting): picked mon.noname-c con 0x7f87d014be90 addr v1:172.21.15.98:6789/0 2024-03-25T18:16:44.942 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.940+0000 7f87d5789640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87d00af940 con 0x7f87d00b4d60 2024-03-25T18:16:44.943 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.940+0000 7f87d5789640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87d00b0a70 con 0x7f87d014be90 2024-03-25T18:16:44.943 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.940+0000 7f87d5789640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87d00b0d50 con 0x7f87d00b1730 2024-03-25T18:16:44.943 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.940+0000 7f87d5789640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:44.943 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.940+0000 7f87d5789640 10 monclient(hunting): authenticate will time out at 7617.215825s 2024-03-25T18:16:44.943 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.941+0000 7f87ce7fc640 1 --1- >> v1:172.21.15.97:6790/0 conn(0x7f87d00b1730 0x7f87d00b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6790/0 says I am v1:172.21.15.97:38150/0 (socket says 172.21.15.97:38150) 2024-03-25T18:16:44.943 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.941+0000 7f87ce7fc640 1 -- 172.21.15.97:0/1127207260 learned_addr learned my addr 172.21.15.97:0/1127207260 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:16:44.944 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.943+0000 7f87cdffb640 1 -- 172.21.15.97:0/1127207260 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3241095161 0 0) 0x7f87d00b0d50 con 0x7f87d00b1730 2024-03-25T18:16:44.944 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.943+0000 7f87cdffb640 10 monclient(hunting): _negotiate 2024-03-25T18:16:44.944 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.944+0000 7f87cdffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:44.944 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.944+0000 7f87cdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:44.944 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.944+0000 7f87cdffb640 10 monclient(hunting): my global_id is 4205 2024-03-25T18:16:44.944 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.944+0000 7f87cdffb640 1 -- 172.21.15.97:0/1127207260 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87b8003520 con 0x7f87d00b1730 2024-03-25T18:16:44.946 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.945+0000 7f87cdffb640 1 -- 172.21.15.97:0/1127207260 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1028313570 0 0) 0x7f87d00b0a70 con 0x7f87d014be90 2024-03-25T18:16:44.946 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.945+0000 7f87cdffb640 10 monclient(hunting): _negotiate 2024-03-25T18:16:44.946 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.945+0000 7f87cdffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:44.946 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.945+0000 7f87cdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:44.946 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.945+0000 7f87cdffb640 10 monclient(hunting): my global_id is 4210 2024-03-25T18:16:44.946 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.945+0000 7f87cdffb640 1 -- 172.21.15.97:0/1127207260 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87d00b0d50 con 0x7f87d014be90 2024-03-25T18:16:44.946 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.945+0000 7f87cdffb640 1 -- 172.21.15.97:0/1127207260 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2879778480 0 0) 0x7f87d00af940 con 0x7f87d00b4d60 2024-03-25T18:16:44.946 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.945+0000 7f87cdffb640 10 monclient(hunting): _negotiate 2024-03-25T18:16:44.946 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.945+0000 7f87cdffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:44.946 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.945+0000 7f87cdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:44.946 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.945+0000 7f87cdffb640 10 monclient(hunting): my global_id is 4212 2024-03-25T18:16:44.946 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.945+0000 7f87cdffb640 1 -- 172.21.15.97:0/1127207260 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87d00b0a70 con 0x7f87d00b4d60 2024-03-25T18:16:44.949 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.948+0000 7f87cdffb640 1 -- 172.21.15.97:0/1127207260 <== mon.2 v1:172.21.15.97:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2050839702 0 0) 0x7f87b8003520 con 0x7f87d00b1730 2024-03-25T18:16:44.950 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.948+0000 7f87cdffb640 1 -- 172.21.15.97:0/1127207260 >> v1:172.21.15.98:6789/0 conn(0x7f87d014be90 legacy=0x7f87d014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:44.950 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.948+0000 7f87cdffb640 1 -- 172.21.15.97:0/1127207260 >> v1:172.21.15.97:6789/0 conn(0x7f87d00b4d60 legacy=0x7f87d013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:44.950 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.948+0000 7f87cdffb640 10 monclient: _finish_hunting 0 2024-03-25T18:16:44.950 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.948+0000 7f87cdffb640 1 monclient: found mon.noname-b 2024-03-25T18:16:44.950 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.948+0000 7f87cdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:44.950 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.948+0000 7f87cdffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.97:6790/0 2024-03-25T18:16:44.950 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.948+0000 7f87cdffb640 1 -- 172.21.15.97:0/1127207260 --> v1:172.21.15.97:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87d00b1030 con 0x7f87d00b1730 2024-03-25T18:16:44.950 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.948+0000 7f87cdffb640 10 monclient: _finish_auth 0 2024-03-25T18:16:44.950 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.948+0000 7f87cdffb640 10 monclient: _check_auth_tickets 2024-03-25T18:16:44.950 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.948+0000 7f87cdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:44.950 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.948+0000 7f87d5789640 5 monclient: authenticate success, global_id 4205 2024-03-25T18:16:44.950 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.948+0000 7f87d5789640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:44.950 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.949+0000 7f87cdffb640 1 -- 172.21.15.97:0/1127207260 <== mon.2 v1:172.21.15.97:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f87c4001650 con 0x7f87d00b1730 2024-03-25T18:16:44.950 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.949+0000 7f87cdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:44.951 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.949+0000 7f87cdffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:16:44.951 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.949+0000 7f87cdffb640 10 monclient: dump: 2024-03-25T18:16:44.951 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:16:44.951 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:16:44.951 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:44.951 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:44.951 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:44.951 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:16:44.951 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:16:44.951 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:16:44.951 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:16:44.951 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:16:44.951 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.949+0000 7f87d5789640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:44.953 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.950+0000 7f87cdffb640 1 -- 172.21.15.97:0/1127207260 <== mon.2 v1:172.21.15.97:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f87c40018a0 con 0x7f87d00b1730 2024-03-25T18:16:44.953 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.950+0000 7f87cdffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:44.953 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.950+0000 7f87cdffb640 1 -- 172.21.15.97:0/1127207260 <== mon.2 v1:172.21.15.97:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f87c4001c30 con 0x7f87d00b1730 2024-03-25T18:16:44.953 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.950+0000 7f87cdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:44.953 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.950+0000 7f87cdffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:16:44.954 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.950+0000 7f87cdffb640 10 monclient: dump: 2024-03-25T18:16:44.954 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:16:44.954 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:16:44.954 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:44.954 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:44.954 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:44.954 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:16:44.954 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:16:44.954 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:16:44.954 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:16:44.954 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:16:44.954 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.950+0000 7f87d5789640 10 monclient: get_monmap_and_config success 2024-03-25T18:16:44.954 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.950+0000 7f87d5789640 4 set_mon_vals no callback set 2024-03-25T18:16:44.954 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.951+0000 7f87d5789640 10 monclient: shutdown 2024-03-25T18:16:44.954 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.951+0000 7f87d5789640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:44.955 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.951+0000 7f87d5789640 1 -- 172.21.15.97:0/1127207260 >> v1:172.21.15.97:6790/0 conn(0x7f87d00b1730 legacy=0x7f87d00b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:44.955 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.951+0000 7f87d5789640 1 -- 172.21.15.97:0/1127207260 shutdown_connections 2024-03-25T18:16:44.955 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.951+0000 7f87d5789640 1 -- 172.21.15.97:0/1127207260 >> 172.21.15.97:0/1127207260 conn(0x7f87d00617a0 msgr2=0x7f87d0061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:44.955 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.951+0000 7f87d5789640 1 -- 172.21.15.97:0/1127207260 shutdown_connections 2024-03-25T18:16:44.955 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.951+0000 7f87d5789640 1 -- 172.21.15.97:0/1127207260 wait complete. 2024-03-25T18:16:44.955 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.952+0000 7f87d5789640 10 monclient: build_initial_monmap 2024-03-25T18:16:44.955 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.952+0000 7f87d5789640 10 monclient: monmap: 2024-03-25T18:16:44.955 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:16:44.955 INFO:teuthology.misc.health.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:16:44.955 INFO:teuthology.misc.health.smithi097.stderr:last_changed 0.000000 2024-03-25T18:16:44.955 INFO:teuthology.misc.health.smithi097.stderr:created 0.000000 2024-03-25T18:16:44.955 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:44.955 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 1 2024-03-25T18:16:44.955 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:16:44.956 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T18:16:44.956 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-c 2024-03-25T18:16:44.956 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:16:44.956 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.952+0000 7f87d5789640 1 librados: starting msgr at 2024-03-25T18:16:44.956 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.952+0000 7f87d5789640 1 librados: starting objecter 2024-03-25T18:16:44.956 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.953+0000 7f87d5789640 1 Processor -- start 2024-03-25T18:16:44.956 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.953+0000 7f87d5789640 1 -- start start 2024-03-25T18:16:44.956 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.953+0000 7f87d5789640 1 librados: setting wanted keys 2024-03-25T18:16:44.956 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.953+0000 7f87d5789640 1 librados: calling monclient init 2024-03-25T18:16:44.956 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.953+0000 7f87d5789640 10 monclient: init 2024-03-25T18:16:44.959 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.954+0000 7f87d5789640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:44.960 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.954+0000 7f87d5789640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:16:44.960 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.954+0000 7f87d5789640 10 monclient(hunting): picked mon.noname-b con 0x7f87d00b1730 addr v1:172.21.15.98:6789/0 2024-03-25T18:16:44.960 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.954+0000 7f87d5789640 10 monclient(hunting): picked mon.noname-c con 0x7f87d00b4d60 addr v1:172.21.15.97:6790/0 2024-03-25T18:16:44.960 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.954+0000 7f87d5789640 10 monclient(hunting): picked mon.noname-a con 0x7f87d014be90 addr v1:172.21.15.97:6789/0 2024-03-25T18:16:44.960 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.954+0000 7f87d5789640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87d0134a20 con 0x7f87d014be90 2024-03-25T18:16:44.960 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.954+0000 7f87d5789640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87d013ae00 con 0x7f87d00b1730 2024-03-25T18:16:44.960 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.954+0000 7f87d5789640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87d013b0e0 con 0x7f87d00b4d60 2024-03-25T18:16:44.960 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.954+0000 7f87ce7fc640 1 --1- >> v1:172.21.15.97:6790/0 conn(0x7f87d00b4d60 0x7f87d0133c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.97:6790/0 says I am v1:172.21.15.97:38152/0 (socket says 172.21.15.97:38152) 2024-03-25T18:16:44.960 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.954+0000 7f87ce7fc640 1 -- 172.21.15.97:0/1205215175 learned_addr learned my addr 172.21.15.97:0/1205215175 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:16:44.960 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.954+0000 7f87d5789640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:44.960 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.954+0000 7f87d5789640 10 monclient(hunting): authenticate will time out at 7617.229749s 2024-03-25T18:16:44.960 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.955+0000 7f87b77fe640 10 client.?.objecter ms_handle_connect 0x7f87d00b4d60 2024-03-25T18:16:44.960 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.955+0000 7f87b77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:44.960 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.955+0000 7f87b77fe640 10 client.?.objecter ms_handle_connect 0x7f87d00b1730 2024-03-25T18:16:44.961 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.955+0000 7f87b77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:44.961 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.955+0000 7f87b77fe640 10 client.?.objecter ms_handle_connect 0x7f87d014be90 2024-03-25T18:16:44.961 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.955+0000 7f87b77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:44.961 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.957+0000 7f87b77fe640 1 -- 172.21.15.97:0/1205215175 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1940267592 0 0) 0x7f87d013b0e0 con 0x7f87d00b4d60 2024-03-25T18:16:44.961 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.957+0000 7f87b77fe640 10 monclient(hunting): _negotiate 2024-03-25T18:16:44.961 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.957+0000 7f87b77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:44.961 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.957+0000 7f87b77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:44.961 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.957+0000 7f87b77fe640 10 monclient(hunting): my global_id is 4208 2024-03-25T18:16:44.961 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.957+0000 7f87b77fe640 1 -- 172.21.15.97:0/1205215175 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87bc004eb0 con 0x7f87d00b4d60 2024-03-25T18:16:44.961 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.958+0000 7f87b77fe640 1 -- 172.21.15.97:0/1205215175 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2175747733 0 0) 0x7f87d013ae00 con 0x7f87d00b1730 2024-03-25T18:16:44.961 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.958+0000 7f87b77fe640 10 monclient(hunting): _negotiate 2024-03-25T18:16:44.961 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.958+0000 7f87b77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:44.961 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.958+0000 7f87b77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:44.961 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.958+0000 7f87b77fe640 10 monclient(hunting): my global_id is 4213 2024-03-25T18:16:44.961 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.958+0000 7f87b77fe640 1 -- 172.21.15.97:0/1205215175 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87d013b0e0 con 0x7f87d00b1730 2024-03-25T18:16:44.962 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.958+0000 7f87b77fe640 1 -- 172.21.15.97:0/1205215175 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1095389512 0 0) 0x7f87d0134a20 con 0x7f87d014be90 2024-03-25T18:16:44.962 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.958+0000 7f87b77fe640 10 monclient(hunting): _negotiate 2024-03-25T18:16:44.962 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.958+0000 7f87b77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:44.962 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.958+0000 7f87b77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:44.962 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.958+0000 7f87b77fe640 10 monclient(hunting): my global_id is 4215 2024-03-25T18:16:44.962 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.958+0000 7f87b77fe640 1 -- 172.21.15.97:0/1205215175 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87d013ae00 con 0x7f87d014be90 2024-03-25T18:16:44.962 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.962+0000 7f87b77fe640 1 -- 172.21.15.97:0/1205215175 <== mon.2 v1:172.21.15.97:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1857903210 0 0) 0x7f87bc004eb0 con 0x7f87d00b4d60 2024-03-25T18:16:44.963 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.962+0000 7f87b77fe640 1 -- 172.21.15.97:0/1205215175 --> v1:172.21.15.97:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f87d0134a20 con 0x7f87d00b4d60 2024-03-25T18:16:44.963 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.962+0000 7f87b77fe640 1 -- 172.21.15.97:0/1205215175 <== mon.2 v1:172.21.15.97:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f87c4001bb0 con 0x7f87d00b4d60 2024-03-25T18:16:44.963 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.962+0000 7f87b77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:44.963 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.962+0000 7f87b77fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:16:44.963 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.962+0000 7f87b77fe640 10 monclient(hunting): dump: 2024-03-25T18:16:44.963 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:16:44.963 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:16:44.963 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:44.963 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:44.963 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:44.963 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:16:44.963 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:16:44.963 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:16:44.963 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:16:44.963 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:16:44.964 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.962+0000 7f87b77fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:16:44.965 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.964+0000 7f87b77fe640 1 -- 172.21.15.97:0/1205215175 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 693799972 0 0) 0x7f87d013ae00 con 0x7f87d014be90 2024-03-25T18:16:44.965 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.964+0000 7f87b77fe640 1 -- 172.21.15.97:0/1205215175 --> v1:172.21.15.97:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f87bc004eb0 con 0x7f87d014be90 2024-03-25T18:16:44.965 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.964+0000 7f87b77fe640 1 -- 172.21.15.97:0/1205215175 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f87c80019f0 con 0x7f87d014be90 2024-03-25T18:16:44.965 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.964+0000 7f87b77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:44.965 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.964+0000 7f87b77fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:16:44.965 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.964+0000 7f87b77fe640 10 monclient(hunting): dump: 2024-03-25T18:16:44.965 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:16:44.965 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:16:44.965 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:44.965 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:44.966 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:44.966 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:16:44.966 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:16:44.966 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:16:44.966 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:16:44.966 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:16:44.966 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.965+0000 7f87b77fe640 1 -- 172.21.15.97:0/1205215175 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2221387211 0 0) 0x7f87d013b0e0 con 0x7f87d00b1730 2024-03-25T18:16:44.966 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.965+0000 7f87b77fe640 1 -- 172.21.15.97:0/1205215175 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f87d013ae00 con 0x7f87d00b1730 2024-03-25T18:16:44.966 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.965+0000 7f87b77fe640 1 -- 172.21.15.97:0/1205215175 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f87c0001870 con 0x7f87d00b1730 2024-03-25T18:16:44.966 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.965+0000 7f87b77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:44.966 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.965+0000 7f87b77fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:16:44.966 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.965+0000 7f87b77fe640 10 monclient(hunting): dump: 2024-03-25T18:16:44.966 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:16:44.966 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:16:44.966 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:44.967 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:44.967 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:44.967 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:16:44.967 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:16:44.967 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:16:44.967 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:16:44.967 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:16:44.969 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.968+0000 7f87b77fe640 1 -- 172.21.15.97:0/1205215175 <== mon.2 v1:172.21.15.97:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 694557556 0 0) 0x7f87d0134a20 con 0x7f87d00b4d60 2024-03-25T18:16:44.970 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.968+0000 7f87b77fe640 1 -- 172.21.15.97:0/1205215175 >> v1:172.21.15.98:6789/0 conn(0x7f87d00b1730 legacy=0x7f87d013b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:44.970 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.968+0000 7f87b77fe640 1 -- 172.21.15.97:0/1205215175 >> v1:172.21.15.97:6789/0 conn(0x7f87d014be90 legacy=0x7f87d0134310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:44.970 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.968+0000 7f87b77fe640 10 monclient: _finish_hunting 0 2024-03-25T18:16:44.970 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.968+0000 7f87b77fe640 1 monclient: found mon.c 2024-03-25T18:16:44.970 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.968+0000 7f87b77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:44.970 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.968+0000 7f87b77fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.97:6790/0 2024-03-25T18:16:44.970 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.968+0000 7f87b77fe640 1 -- 172.21.15.97:0/1205215175 --> v1:172.21.15.97:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87d0139bd0 con 0x7f87d00b4d60 2024-03-25T18:16:44.970 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.968+0000 7f87b77fe640 10 monclient: _finish_auth 0 2024-03-25T18:16:44.970 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.968+0000 7f87b77fe640 10 monclient: _check_auth_tickets 2024-03-25T18:16:44.970 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.968+0000 7f87b77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:44.970 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.968+0000 7f87d5789640 5 monclient: authenticate success, global_id 4208 2024-03-25T18:16:44.970 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.968+0000 7f87d5789640 10 monclient: _renew_subs 2024-03-25T18:16:44.970 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.968+0000 7f87d5789640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.97:6790/0 2024-03-25T18:16:44.970 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.968+0000 7f87d5789640 1 -- 172.21.15.97:0/1205215175 --> v1:172.21.15.97:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87d0139f60 con 0x7f87d00b4d60 2024-03-25T18:16:44.971 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.968+0000 7f87d5789640 10 client.4208.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:16:44.971 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.968+0000 7f87d5789640 10 monclient: _renew_subs 2024-03-25T18:16:44.971 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.968+0000 7f87d5789640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.97:6790/0 2024-03-25T18:16:44.971 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.968+0000 7f87d5789640 1 -- 172.21.15.97:0/1205215175 --> v1:172.21.15.97:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f87d0143130 con 0x7f87d00b4d60 2024-03-25T18:16:44.971 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.968+0000 7f87d5789640 1 librados: init done 2024-03-25T18:16:44.971 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.970+0000 7f87b77fe640 1 -- 172.21.15.97:0/1205215175 <== mon.2 v1:172.21.15.97:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f87c4004bf0 con 0x7f87d00b4d60 2024-03-25T18:16:44.971 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.970+0000 7f87b77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:44.971 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.970+0000 7f87b77fe640 1 -- 172.21.15.97:0/1205215175 <== mon.2 v1:172.21.15.97:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f87c4004f00 con 0x7f87d00b4d60 2024-03-25T18:16:44.971 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.970+0000 7f87b77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:44.971 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.970+0000 7f87b77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:16:44.971 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.970+0000 7f87b77fe640 10 monclient: dump: 2024-03-25T18:16:44.971 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:16:44.971 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:16:44.971 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:44.971 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:44.972 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:44.972 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:16:44.972 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:16:44.972 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:16:44.972 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:16:44.972 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:16:44.973 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.970+0000 7f87d5789640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:44.973 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.970+0000 7f87d5789640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:44.973 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.970+0000 7f87d5789640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.97:6790/0 2024-03-25T18:16:44.973 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.970+0000 7f87cd7fa640 4 set_mon_vals no callback set 2024-03-25T18:16:44.973 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:44.970+0000 7f87d5789640 1 -- 172.21.15.97:0/1205215175 --> v1:172.21.15.97:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87d0140240 con 0x7f87d00b4d60 2024-03-25T18:16:45.341 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:45.340+0000 7f87b77fe640 1 -- 172.21.15.97:0/1205215175 <== mon.2 v1:172.21.15.97:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2377622137 0 0) 0x7f87c401aed0 con 0x7f87d00b4d60 2024-03-25T18:16:45.341 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:45.341+0000 7f87ceffd640 10 monclient: get_auth_request con 0x7f87bc06e0a0 auth_method 0 2024-03-25T18:16:45.341 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:45.341+0000 7f87b77fe640 10 client.4208.objecter ms_handle_connect 0x7f87bc06e0a0 2024-03-25T18:16:45.345 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:45.345+0000 7f87b77fe640 1 -- 172.21.15.97:0/1205215175 <== mon.2 v1:172.21.15.97:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 430711918 0 0) 0x7f87c4054a10 con 0x7f87d00b4d60 2024-03-25T18:16:45.345 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:45.345+0000 7f87b77fe640 10 client.4208.objecter ms_dispatch 0x7f87d0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:16:45.345 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:45.345+0000 7f87b77fe640 3 client.4208.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:16:45.345 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:45.345+0000 7f87b77fe640 3 client.4208.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:16:45.345 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:45.345+0000 7f87b77fe640 20 client.4208.objecter dump_active .. 0 homeless 2024-03-25T18:16:45.345 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:45.345+0000 7f87b77fe640 1 -- 172.21.15.97:0/1205215175 <== mon.2 v1:172.21.15.97:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f87c4050350 con 0x7f87d00b4d60 2024-03-25T18:16:45.346 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:45.345+0000 7f87b77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:45.346 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:45.345+0000 7f87b77fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:16:45.568 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:45.568+0000 7f87d5789640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:45.568 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:45.568+0000 7f87d5789640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:45.568 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:45.568+0000 7f87d5789640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.97:6790/0 2024-03-25T18:16:45.568 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:45.568+0000 7f87d5789640 1 -- 172.21.15.97:0/1205215175 --> v1:172.21.15.97:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f87d0136d90 con 0x7f87d00b4d60 2024-03-25T18:16:45.592 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:45.592+0000 7f87b77fe640 1 -- 172.21.15.97:0/1205215175 <== mon.2 v1:172.21.15.97:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2169427412) 0x7f87c4052450 con 0x7f87d00b4d60 2024-03-25T18:16:45.592 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:45.592+0000 7f87b77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:45.592 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:45.592+0000 7f87b77fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:16:45.593 INFO:teuthology.misc.health.smithi097.stdout: 2024-03-25T18:16:45.593 INFO:teuthology.misc.health.smithi097.stdout:{"epoch":10,"fsid":"4c3d472c-79b8-486b-9e3d-2738d4306754","created":"2024-03-25T18:16:14.737309+0000","modified":"2024-03-25T18:16:20.357492+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:20.357492+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cb5152b9-262f-45c3-94b8-18d0e81df558","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6daf8793-0838-473d-a7d6-9a0d2894c75d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"136f1493-3b9c-4f43-a2e7-641482d69dae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"397d538c-4f64-4daf-9e8b-662d09f91327","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4f1df08-663f-4bda-93f8-3878ab7c24b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c4b18d70-a3ce-4b15-9b70-eae5525bd761","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8dd4033a-de0e-4f8a-96dc-59f3c1f825b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0077722e-ffa6-4a3d-9891-929d9869b286","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:16:45.594 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:45.594+0000 7f87d5789640 10 librados: watch_flush enter 2024-03-25T18:16:45.594 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:45.594+0000 7f87d5789640 10 librados: watch_flush exit 2024-03-25T18:16:45.594 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:45.594+0000 7f87d5789640 20 client.4208.objecter shutdown clearing up homeless session... 2024-03-25T18:16:45.595 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:45.594+0000 7f87d5789640 10 client.4208.objecter successfully canceled tick 2024-03-25T18:16:45.596 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:45.596+0000 7f87d5789640 1 -- 172.21.15.97:0/1205215175 >> v1:172.21.15.98:6800/800387573 conn(0x7f87bc06e0a0 legacy=0x7f87bc070560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:45.597 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:45.596+0000 7f87d5789640 10 monclient: shutdown 2024-03-25T18:16:45.597 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:45.596+0000 7f87d5789640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:45.597 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:45.596+0000 7f87d5789640 1 -- 172.21.15.97:0/1205215175 >> v1:172.21.15.97:6790/0 conn(0x7f87d00b4d60 legacy=0x7f87d0133c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:45.597 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:45.596+0000 7f87d5789640 1 -- 172.21.15.97:0/1205215175 shutdown_connections 2024-03-25T18:16:45.597 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:45.596+0000 7f87d5789640 1 -- 172.21.15.97:0/1205215175 >> 172.21.15.97:0/1205215175 conn(0x7f87d00617a0 msgr2=0x7f87d0137310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:45.597 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:45.596+0000 7f87d5789640 1 -- 172.21.15.97:0/1205215175 shutdown_connections 2024-03-25T18:16:45.597 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:45.596+0000 7f87d5789640 1 -- 172.21.15.97:0/1205215175 wait complete. 2024-03-25T18:16:45.597 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:45.596+0000 7f87d5789640 1 librados: shutdown 2024-03-25T18:16:45.607 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:16:46.402 INFO:tasks.ceph.osd.7.smithi098.stderr:==00:00:00:25.880 35255== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:46.407 INFO:tasks.ceph.osd.5.smithi098.stderr:==00:00:00:25.912 35253== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:46.576 INFO:tasks.ceph.osd.4.smithi098.stderr:==00:00:00:26.082 35252== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:46.708 INFO:tasks.ceph.osd.6.smithi098.stderr:==00:00:00:26.206 35254== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:47.289 INFO:tasks.ceph.osd.0.smithi097.stderr:==00:00:00:26.797 35245== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:47.870 INFO:tasks.ceph.osd.2.smithi097.stderr:==00:00:00:27.371 35247== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:48.548 INFO:tasks.ceph.osd.3.smithi097.stderr:==00:00:00:28.056 35246== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:49.046 INFO:tasks.ceph.osd.1.smithi097.stderr:==00:00:00:28.535 35248== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:52.412 DEBUG:teuthology.orchestra.run.smithi097:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:16:52.556 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.556+0000 7fcee8ac8640 10 monclient: get_monmap_and_config 2024-03-25T18:16:52.556 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.556+0000 7fcee8ac8640 10 monclient: build_initial_monmap 2024-03-25T18:16:52.556 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.556+0000 7fcee8ac8640 10 monclient: monmap: 2024-03-25T18:16:52.556 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:16:52.556 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:16:52.556 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:16:52.556648+0000 2024-03-25T18:16:52.556 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:16:52.556648+0000 2024-03-25T18:16:52.557 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:52.557 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:16:52.557 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:16:52.557 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T18:16:52.557 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-b 2024-03-25T18:16:52.557 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:16:52.557 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.557+0000 7fcee8ac8640 1 Processor -- start 2024-03-25T18:16:52.557 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.557+0000 7fcee8ac8640 1 -- start start 2024-03-25T18:16:52.557 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.557+0000 7fcee8ac8640 10 monclient: init 2024-03-25T18:16:52.559 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.558+0000 7fcee8ac8640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:52.559 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.558+0000 7fcee8ac8640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:16:52.559 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.558+0000 7fcee8ac8640 10 monclient(hunting): picked mon.noname-b con 0x7fcee40b4d60 addr v1:172.21.15.97:6790/0 2024-03-25T18:16:52.559 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.558+0000 7fcee8ac8640 10 monclient(hunting): picked mon.noname-a con 0x7fcee40b1730 addr v1:172.21.15.97:6789/0 2024-03-25T18:16:52.559 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.558+0000 7fcee8ac8640 10 monclient(hunting): picked mon.noname-c con 0x7fcee414be90 addr v1:172.21.15.98:6789/0 2024-03-25T18:16:52.559 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.558+0000 7fcee8ac8640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcee40af940 con 0x7fcee40b1730 2024-03-25T18:16:52.559 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.558+0000 7fcee8ac8640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcee40b0a70 con 0x7fcee414be90 2024-03-25T18:16:52.559 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.559+0000 7fcee8ac8640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcee40b0d50 con 0x7fcee40b4d60 2024-03-25T18:16:52.559 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.559+0000 7fcee8ac8640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:52.559 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.559+0000 7fcee8ac8640 10 monclient(hunting): authenticate will time out at 7624.834034s 2024-03-25T18:16:52.560 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.559+0000 7fcee2d9d640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fcee414be90 0x7fcee414e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.97:33994/0 (socket says 172.21.15.97:33994) 2024-03-25T18:16:52.560 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.559+0000 7fcee2d9d640 1 -- 172.21.15.97:0/568957896 learned_addr learned my addr 172.21.15.97:0/568957896 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:16:52.564 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.563+0000 7fcee1d9b640 1 -- 172.21.15.97:0/568957896 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 375934192 0 0) 0x7fcee40b0a70 con 0x7fcee414be90 2024-03-25T18:16:52.564 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.563+0000 7fcee1d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:16:52.564 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.563+0000 7fcee1d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:52.564 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.563+0000 7fcee1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:52.564 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.563+0000 7fcee1d9b640 10 monclient(hunting): my global_id is 4216 2024-03-25T18:16:52.564 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.563+0000 7fcee1d9b640 1 -- 172.21.15.97:0/568957896 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fced0003520 con 0x7fcee414be90 2024-03-25T18:16:52.564 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.563+0000 7fcee1d9b640 1 -- 172.21.15.97:0/568957896 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 852946659 0 0) 0x7fcee40b0d50 con 0x7fcee40b4d60 2024-03-25T18:16:52.564 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.563+0000 7fcee1d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:16:52.565 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.563+0000 7fcee1d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:52.565 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.563+0000 7fcee1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:52.565 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.563+0000 7fcee1d9b640 10 monclient(hunting): my global_id is 4211 2024-03-25T18:16:52.565 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.563+0000 7fcee1d9b640 1 -- 172.21.15.97:0/568957896 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcee40b0a70 con 0x7fcee40b4d60 2024-03-25T18:16:52.565 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.563+0000 7fcee1d9b640 1 -- 172.21.15.97:0/568957896 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1666830906 0 0) 0x7fcee40af940 con 0x7fcee40b1730 2024-03-25T18:16:52.565 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.563+0000 7fcee1d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:16:52.565 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.563+0000 7fcee1d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:52.565 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.563+0000 7fcee1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:52.565 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.563+0000 7fcee1d9b640 10 monclient(hunting): my global_id is 4218 2024-03-25T18:16:52.565 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.563+0000 7fcee1d9b640 1 -- 172.21.15.97:0/568957896 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcee40b0d50 con 0x7fcee40b1730 2024-03-25T18:16:52.568 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.567+0000 7fcee1d9b640 1 -- 172.21.15.97:0/568957896 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1761249083 0 0) 0x7fcee40b0d50 con 0x7fcee40b1730 2024-03-25T18:16:52.568 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.567+0000 7fcee1d9b640 1 -- 172.21.15.97:0/568957896 >> v1:172.21.15.97:6790/0 conn(0x7fcee40b4d60 legacy=0x7fcee413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:52.568 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.567+0000 7fcee1d9b640 1 -- 172.21.15.97:0/568957896 >> v1:172.21.15.98:6789/0 conn(0x7fcee414be90 legacy=0x7fcee414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:52.568 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.567+0000 7fcee1d9b640 10 monclient: _finish_hunting 0 2024-03-25T18:16:52.568 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.567+0000 7fcee1d9b640 1 monclient: found mon.noname-a 2024-03-25T18:16:52.568 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.567+0000 7fcee1d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:52.568 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.567+0000 7fcee1d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.97:6789/0 2024-03-25T18:16:52.568 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.567+0000 7fcee1d9b640 1 -- 172.21.15.97:0/568957896 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcee40b1030 con 0x7fcee40b1730 2024-03-25T18:16:52.568 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.567+0000 7fcee1d9b640 10 monclient: _finish_auth 0 2024-03-25T18:16:52.569 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.567+0000 7fcee1d9b640 10 monclient: _check_auth_tickets 2024-03-25T18:16:52.569 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.567+0000 7fcee1d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:52.569 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.567+0000 7fcee1d9b640 1 -- 172.21.15.97:0/568957896 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7fcecc0015b0 con 0x7fcee40b1730 2024-03-25T18:16:52.569 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.567+0000 7fcee8ac8640 5 monclient: authenticate success, global_id 4218 2024-03-25T18:16:52.569 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.567+0000 7fcee1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:52.569 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.567+0000 7fcee1d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:16:52.569 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.567+0000 7fcee1d9b640 10 monclient: dump: 2024-03-25T18:16:52.569 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:16:52.569 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:16:52.569 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:52.569 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:52.569 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:52.569 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:16:52.569 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:16:52.569 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:16:52.570 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:16:52.570 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:16:52.570 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.567+0000 7fcee8ac8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:52.571 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.568+0000 7fcee1d9b640 1 -- 172.21.15.97:0/568957896 <== mon.0 v1:172.21.15.97:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcecc001710 con 0x7fcee40b1730 2024-03-25T18:16:52.571 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.568+0000 7fcee1d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:52.571 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.568+0000 7fcee1d9b640 1 -- 172.21.15.97:0/568957896 <== mon.0 v1:172.21.15.97:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7fcecc001b60 con 0x7fcee40b1730 2024-03-25T18:16:52.571 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.568+0000 7fcee1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:52.571 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.568+0000 7fcee1d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:16:52.571 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.568+0000 7fcee1d9b640 10 monclient: dump: 2024-03-25T18:16:52.571 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:16:52.571 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:16:52.572 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:52.572 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:52.572 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:52.572 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:16:52.572 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:16:52.572 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:16:52.572 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:16:52.572 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:16:52.572 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.568+0000 7fcee8ac8640 10 monclient: get_monmap_and_config success 2024-03-25T18:16:52.572 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.569+0000 7fcee8ac8640 4 set_mon_vals no callback set 2024-03-25T18:16:52.572 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.570+0000 7fcee8ac8640 10 monclient: shutdown 2024-03-25T18:16:52.572 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.570+0000 7fcee8ac8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:52.572 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.570+0000 7fcee8ac8640 1 -- 172.21.15.97:0/568957896 >> v1:172.21.15.97:6789/0 conn(0x7fcee40b1730 legacy=0x7fcee40b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:52.572 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.570+0000 7fcee8ac8640 1 -- 172.21.15.97:0/568957896 shutdown_connections 2024-03-25T18:16:52.572 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.570+0000 7fcee8ac8640 1 -- 172.21.15.97:0/568957896 >> 172.21.15.97:0/568957896 conn(0x7fcee40617a0 msgr2=0x7fcee4061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:52.573 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.570+0000 7fcee8ac8640 1 -- 172.21.15.97:0/568957896 shutdown_connections 2024-03-25T18:16:52.573 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.570+0000 7fcee8ac8640 1 -- 172.21.15.97:0/568957896 wait complete. 2024-03-25T18:16:52.574 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.571+0000 7fcee8ac8640 10 monclient: build_initial_monmap 2024-03-25T18:16:52.574 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.571+0000 7fcee8ac8640 10 monclient: monmap: 2024-03-25T18:16:52.575 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:16:52.575 INFO:teuthology.misc.health.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:16:52.575 INFO:teuthology.misc.health.smithi097.stderr:last_changed 0.000000 2024-03-25T18:16:52.575 INFO:teuthology.misc.health.smithi097.stderr:created 0.000000 2024-03-25T18:16:52.575 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:52.575 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 1 2024-03-25T18:16:52.575 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:16:52.575 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T18:16:52.575 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-c 2024-03-25T18:16:52.575 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:16:52.575 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.572+0000 7fcee8ac8640 1 librados: starting msgr at 2024-03-25T18:16:52.575 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.572+0000 7fcee8ac8640 1 librados: starting objecter 2024-03-25T18:16:52.575 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.572+0000 7fcee8ac8640 1 Processor -- start 2024-03-25T18:16:52.575 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.572+0000 7fcee8ac8640 1 -- start start 2024-03-25T18:16:52.575 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.572+0000 7fcee8ac8640 1 librados: setting wanted keys 2024-03-25T18:16:52.575 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.572+0000 7fcee8ac8640 1 librados: calling monclient init 2024-03-25T18:16:52.576 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.572+0000 7fcee8ac8640 10 monclient: init 2024-03-25T18:16:52.576 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.572+0000 7fcee8ac8640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:52.576 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.572+0000 7fcee8ac8640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:16:52.576 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.572+0000 7fcee8ac8640 10 monclient(hunting): picked mon.noname-a con 0x7fcee40b1730 addr v1:172.21.15.97:6789/0 2024-03-25T18:16:52.576 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.572+0000 7fcee8ac8640 10 monclient(hunting): picked mon.noname-b con 0x7fcee40b4d60 addr v1:172.21.15.98:6789/0 2024-03-25T18:16:52.576 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.572+0000 7fcee8ac8640 10 monclient(hunting): picked mon.noname-c con 0x7fcee414be90 addr v1:172.21.15.97:6790/0 2024-03-25T18:16:52.576 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.572+0000 7fcee8ac8640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcee4143900 con 0x7fcee40b1730 2024-03-25T18:16:52.576 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.572+0000 7fcee8ac8640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcee4143a90 con 0x7fcee40b4d60 2024-03-25T18:16:52.576 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.572+0000 7fcee8ac8640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcee41db8e0 con 0x7fcee414be90 2024-03-25T18:16:52.576 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.572+0000 7fcee8ac8640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:52.576 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.572+0000 7fcee8ac8640 10 monclient(hunting): authenticate will time out at 7624.847692s 2024-03-25T18:16:52.576 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.573+0000 7fcee259c640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fcee40b1730 0x7fcee4135c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:38782/0 (socket says 172.21.15.97:38782) 2024-03-25T18:16:52.576 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.573+0000 7fcee259c640 1 -- 172.21.15.97:0/3220407326 learned_addr learned my addr 172.21.15.97:0/3220407326 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:16:52.576 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.573+0000 7fcedaffd640 10 client.?.objecter ms_handle_connect 0x7fcee40b1730 2024-03-25T18:16:52.577 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.573+0000 7fcedaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:52.577 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.573+0000 7fcedaffd640 10 client.?.objecter ms_handle_connect 0x7fcee414be90 2024-03-25T18:16:52.577 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.573+0000 7fcedaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:52.577 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.574+0000 7fcedaffd640 10 client.?.objecter ms_handle_connect 0x7fcee40b4d60 2024-03-25T18:16:52.577 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.574+0000 7fcedaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:52.579 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.575+0000 7fcedaffd640 1 -- 172.21.15.97:0/3220407326 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1276673021 0 0) 0x7fcee41db8e0 con 0x7fcee414be90 2024-03-25T18:16:52.579 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.575+0000 7fcedaffd640 10 monclient(hunting): _negotiate 2024-03-25T18:16:52.579 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.575+0000 7fcedaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:52.579 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.575+0000 7fcedaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:52.579 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.575+0000 7fcedaffd640 10 monclient(hunting): my global_id is 4214 2024-03-25T18:16:52.579 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.576+0000 7fcedaffd640 1 -- 172.21.15.97:0/3220407326 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcec4004eb0 con 0x7fcee414be90 2024-03-25T18:16:52.580 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.576+0000 7fcedaffd640 1 -- 172.21.15.97:0/3220407326 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3284724082 0 0) 0x7fcee4143900 con 0x7fcee40b1730 2024-03-25T18:16:52.580 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.576+0000 7fcedaffd640 10 monclient(hunting): _negotiate 2024-03-25T18:16:52.580 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.576+0000 7fcedaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:52.580 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.576+0000 7fcedaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:52.580 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.577+0000 7fcedaffd640 10 monclient(hunting): my global_id is 4221 2024-03-25T18:16:52.580 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.577+0000 7fcedaffd640 1 -- 172.21.15.97:0/3220407326 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcee41db8e0 con 0x7fcee40b1730 2024-03-25T18:16:52.580 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.577+0000 7fcedaffd640 1 -- 172.21.15.97:0/3220407326 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3148287687 0 0) 0x7fcee4143a90 con 0x7fcee40b4d60 2024-03-25T18:16:52.580 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.577+0000 7fcedaffd640 10 monclient(hunting): _negotiate 2024-03-25T18:16:52.580 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.577+0000 7fcedaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:52.580 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.577+0000 7fcedaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:52.580 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.577+0000 7fcedaffd640 10 monclient(hunting): my global_id is 4219 2024-03-25T18:16:52.581 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.577+0000 7fcedaffd640 1 -- 172.21.15.97:0/3220407326 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcee4143900 con 0x7fcee40b4d60 2024-03-25T18:16:52.581 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.580+0000 7fcedaffd640 1 -- 172.21.15.97:0/3220407326 <== mon.2 v1:172.21.15.97:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3233940002 0 0) 0x7fcec4004eb0 con 0x7fcee414be90 2024-03-25T18:16:52.581 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.580+0000 7fcedaffd640 1 -- 172.21.15.97:0/3220407326 --> v1:172.21.15.97:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcee4143a90 con 0x7fcee414be90 2024-03-25T18:16:52.582 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.580+0000 7fcedaffd640 1 -- 172.21.15.97:0/3220407326 <== mon.2 v1:172.21.15.97:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7fcedc0017b0 con 0x7fcee414be90 2024-03-25T18:16:52.582 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.580+0000 7fcedaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:52.582 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.580+0000 7fcedaffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:16:52.582 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.580+0000 7fcedaffd640 10 monclient(hunting): dump: 2024-03-25T18:16:52.582 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:16:52.582 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:16:52.582 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:52.582 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:52.582 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:52.582 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:16:52.582 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:16:52.582 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:16:52.582 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:16:52.582 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:16:52.583 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.580+0000 7fcedaffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:16:52.583 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.583+0000 7fcedaffd640 1 -- 172.21.15.97:0/3220407326 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 623023064 0 0) 0x7fcee4143900 con 0x7fcee40b4d60 2024-03-25T18:16:52.584 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.584+0000 7fcedaffd640 1 -- 172.21.15.97:0/3220407326 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcec4004eb0 con 0x7fcee40b4d60 2024-03-25T18:16:52.585 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.584+0000 7fcedaffd640 1 -- 172.21.15.97:0/3220407326 <== mon.2 v1:172.21.15.97:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2405734285 0 0) 0x7fcee4143a90 con 0x7fcee414be90 2024-03-25T18:16:52.586 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.584+0000 7fcedaffd640 1 -- 172.21.15.97:0/3220407326 >> v1:172.21.15.98:6789/0 conn(0x7fcee40b4d60 legacy=0x7fcee4136970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:52.586 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.584+0000 7fcedaffd640 1 -- 172.21.15.97:0/3220407326 >> v1:172.21.15.97:6789/0 conn(0x7fcee40b1730 legacy=0x7fcee4135c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:52.586 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.584+0000 7fcedaffd640 10 monclient: _finish_hunting 0 2024-03-25T18:16:52.586 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.584+0000 7fcedaffd640 1 monclient: found mon.c 2024-03-25T18:16:52.586 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.584+0000 7fcedaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:52.586 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.584+0000 7fcedaffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.97:6790/0 2024-03-25T18:16:52.586 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.584+0000 7fcedaffd640 1 -- 172.21.15.97:0/3220407326 --> v1:172.21.15.97:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcee41dba70 con 0x7fcee414be90 2024-03-25T18:16:52.586 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.585+0000 7fcedaffd640 10 monclient: _finish_auth 0 2024-03-25T18:16:52.586 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.585+0000 7fcedaffd640 10 monclient: _check_auth_tickets 2024-03-25T18:16:52.586 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.585+0000 7fcedaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:52.586 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.585+0000 7fcee8ac8640 5 monclient: authenticate success, global_id 4214 2024-03-25T18:16:52.586 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.585+0000 7fcee8ac8640 10 monclient: _renew_subs 2024-03-25T18:16:52.586 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.585+0000 7fcee8ac8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.97:6790/0 2024-03-25T18:16:52.586 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.585+0000 7fcee8ac8640 1 -- 172.21.15.97:0/3220407326 --> v1:172.21.15.97:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcee4137080 con 0x7fcee414be90 2024-03-25T18:16:52.587 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.585+0000 7fcee8ac8640 10 client.4214.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:16:52.587 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.585+0000 7fcee8ac8640 10 monclient: _renew_subs 2024-03-25T18:16:52.587 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.585+0000 7fcee8ac8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.97:6790/0 2024-03-25T18:16:52.587 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.585+0000 7fcee8ac8640 1 -- 172.21.15.97:0/3220407326 --> v1:172.21.15.97:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcee413bd80 con 0x7fcee414be90 2024-03-25T18:16:52.587 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.585+0000 7fcee8ac8640 1 librados: init done 2024-03-25T18:16:52.587 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.586+0000 7fcee8ac8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:52.587 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.586+0000 7fcee8ac8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:52.588 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.586+0000 7fcee8ac8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.97:6790/0 2024-03-25T18:16:52.588 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.586+0000 7fcee8ac8640 1 -- 172.21.15.97:0/3220407326 --> v1:172.21.15.97:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcee40627b0 con 0x7fcee414be90 2024-03-25T18:16:52.601 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.601+0000 7fcedaffd640 1 -- 172.21.15.97:0/3220407326 <== mon.2 v1:172.21.15.97:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcedc001d60 con 0x7fcee414be90 2024-03-25T18:16:52.602 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.601+0000 7fcedaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:52.602 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.601+0000 7fcedaffd640 1 -- 172.21.15.97:0/3220407326 <== mon.2 v1:172.21.15.97:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7fcedc004980 con 0x7fcee414be90 2024-03-25T18:16:52.602 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.601+0000 7fcedaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:52.602 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.601+0000 7fcedaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:16:52.602 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.601+0000 7fcedaffd640 10 monclient: dump: 2024-03-25T18:16:52.602 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:16:52.602 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:16:52.602 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:52.602 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:16:52.602 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:52.602 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:16:52.602 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:16:52.602 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:16:52.603 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:16:52.603 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:16:52.603 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.601+0000 7fcee159a640 4 set_mon_vals no callback set 2024-03-25T18:16:52.603 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.602+0000 7fcedaffd640 1 -- 172.21.15.97:0/3220407326 <== mon.2 v1:172.21.15.97:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2377622137 0 0) 0x7fcedc01a950 con 0x7fcee414be90 2024-03-25T18:16:52.603 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.603+0000 7fcee259c640 10 monclient: get_auth_request con 0x7fcec406de70 auth_method 0 2024-03-25T18:16:52.603 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.603+0000 7fcedaffd640 10 client.4214.objecter ms_handle_connect 0x7fcec406de70 2024-03-25T18:16:52.634 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.633+0000 7fcedaffd640 1 -- 172.21.15.97:0/3220407326 <== mon.2 v1:172.21.15.97:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 430711918 0 0) 0x7fcedc054590 con 0x7fcee414be90 2024-03-25T18:16:52.634 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.633+0000 7fcedaffd640 10 client.4214.objecter ms_dispatch 0x7fcee4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:16:52.634 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.634+0000 7fcedaffd640 3 client.4214.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:16:52.634 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.634+0000 7fcedaffd640 3 client.4214.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:16:52.634 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.634+0000 7fcedaffd640 20 client.4214.objecter dump_active .. 0 homeless 2024-03-25T18:16:52.989 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.989+0000 7fcedaffd640 1 -- 172.21.15.97:0/3220407326 <== mon.2 v1:172.21.15.97:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fcedc04fed0 con 0x7fcee414be90 2024-03-25T18:16:52.989 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.989+0000 7fcedaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:52.989 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:52.989+0000 7fcedaffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:16:53.210 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:53.210+0000 7fcee8ac8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:53.210 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:53.210+0000 7fcee8ac8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:53.210 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:53.210+0000 7fcee8ac8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.97:6790/0 2024-03-25T18:16:53.210 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:53.210+0000 7fcee8ac8640 1 -- 172.21.15.97:0/3220407326 --> v1:172.21.15.97:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcee4136510 con 0x7fcee414be90 2024-03-25T18:16:53.227 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:53.227+0000 7fcedaffd640 1 -- 172.21.15.97:0/3220407326 <== mon.2 v1:172.21.15.97:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2169427412) 0x7fcedc051fd0 con 0x7fcee414be90 2024-03-25T18:16:53.227 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:53.227+0000 7fcedaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:53.227 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:53.227+0000 7fcedaffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:16:53.228 INFO:teuthology.misc.health.smithi097.stdout: 2024-03-25T18:16:53.228 INFO:teuthology.misc.health.smithi097.stdout:{"epoch":10,"fsid":"4c3d472c-79b8-486b-9e3d-2738d4306754","created":"2024-03-25T18:16:14.737309+0000","modified":"2024-03-25T18:16:20.357492+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:20.357492+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cb5152b9-262f-45c3-94b8-18d0e81df558","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6daf8793-0838-473d-a7d6-9a0d2894c75d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"136f1493-3b9c-4f43-a2e7-641482d69dae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"397d538c-4f64-4daf-9e8b-662d09f91327","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4f1df08-663f-4bda-93f8-3878ab7c24b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c4b18d70-a3ce-4b15-9b70-eae5525bd761","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8dd4033a-de0e-4f8a-96dc-59f3c1f825b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0077722e-ffa6-4a3d-9891-929d9869b286","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:16:53.230 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:53.230+0000 7fcee8ac8640 10 librados: watch_flush enter 2024-03-25T18:16:53.230 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:53.230+0000 7fcee8ac8640 10 librados: watch_flush exit 2024-03-25T18:16:53.230 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:53.230+0000 7fcee8ac8640 20 client.4214.objecter shutdown clearing up homeless session... 2024-03-25T18:16:53.230 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:53.230+0000 7fcee8ac8640 10 client.4214.objecter successfully canceled tick 2024-03-25T18:16:53.231 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:53.230+0000 7fcee8ac8640 1 -- 172.21.15.97:0/3220407326 >> v1:172.21.15.98:6800/800387573 conn(0x7fcec406de70 legacy=0x7fcec4070330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:53.231 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:53.230+0000 7fcee8ac8640 10 monclient: shutdown 2024-03-25T18:16:53.231 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:53.230+0000 7fcee8ac8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:53.231 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:53.230+0000 7fcee8ac8640 1 -- 172.21.15.97:0/3220407326 >> v1:172.21.15.97:6790/0 conn(0x7fcee414be90 legacy=0x7fcee413b490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:53.231 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:53.231+0000 7fcee8ac8640 1 -- 172.21.15.97:0/3220407326 shutdown_connections 2024-03-25T18:16:53.231 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:53.231+0000 7fcee8ac8640 1 -- 172.21.15.97:0/3220407326 >> 172.21.15.97:0/3220407326 conn(0x7fcee40617a0 msgr2=0x7fcee413c700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:53.231 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:53.231+0000 7fcee8ac8640 1 -- 172.21.15.97:0/3220407326 shutdown_connections 2024-03-25T18:16:53.231 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:53.231+0000 7fcee8ac8640 1 -- 172.21.15.97:0/3220407326 wait complete. 2024-03-25T18:16:53.232 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:16:53.231+0000 7fcee8ac8640 1 librados: shutdown 2024-03-25T18:16:53.245 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:16:54.929 INFO:tasks.ceph.osd.4.smithi098.stderr:2024-03-25T18:16:54.925+0000 5dc3480 -1 osd.4 0 log_to_monitors true 2024-03-25T18:16:55.092 INFO:tasks.ceph.osd.7.smithi098.stderr:2024-03-25T18:16:55.088+0000 5dc3480 -1 osd.7 0 log_to_monitors true 2024-03-25T18:16:55.120 INFO:tasks.ceph.osd.5.smithi098.stderr:2024-03-25T18:16:55.117+0000 5dc3480 -1 osd.5 0 log_to_monitors true 2024-03-25T18:16:55.413 INFO:tasks.ceph.osd.6.smithi098.stderr:2024-03-25T18:16:55.408+0000 5dc3480 -1 osd.6 0 log_to_monitors true 2024-03-25T18:16:55.847 INFO:tasks.ceph.osd.0.smithi097.stderr:2024-03-25T18:16:55.844+0000 5dc3480 -1 osd.0 0 log_to_monitors true 2024-03-25T18:16:57.000 INFO:tasks.ceph.osd.2.smithi097.stderr:2024-03-25T18:16:56.995+0000 5dc3480 -1 osd.2 0 log_to_monitors true 2024-03-25T18:16:57.542 INFO:tasks.ceph.osd.3.smithi097.stderr:2024-03-25T18:16:57.537+0000 5dc3480 -1 osd.3 0 log_to_monitors true 2024-03-25T18:16:58.508 INFO:tasks.ceph.osd.1.smithi097.stderr:2024-03-25T18:16:58.503+0000 5dc3480 -1 osd.1 0 log_to_monitors true 2024-03-25T18:17:00.052 DEBUG:teuthology.orchestra.run.smithi097:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:17:00.172 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.170+0000 7fe67cb5d640 10 monclient: get_monmap_and_config 2024-03-25T18:17:00.172 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.170+0000 7fe67cb5d640 10 monclient: build_initial_monmap 2024-03-25T18:17:00.172 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.170+0000 7fe67cb5d640 10 monclient: monmap: 2024-03-25T18:17:00.172 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:17:00.172 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:17:00.172 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:17:00.172019+0000 2024-03-25T18:17:00.172 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:17:00.172019+0000 2024-03-25T18:17:00.172 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:00.172 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:17:00.172 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:17:00.173 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T18:17:00.173 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-b 2024-03-25T18:17:00.173 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:17:00.174 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.171+0000 7fe67cb5d640 1 Processor -- start 2024-03-25T18:17:00.174 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.171+0000 7fe67cb5d640 1 -- start start 2024-03-25T18:17:00.174 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.171+0000 7fe67cb5d640 10 monclient: init 2024-03-25T18:17:00.174 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.172+0000 7fe67cb5d640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:00.174 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.172+0000 7fe67cb5d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:17:00.174 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.172+0000 7fe67cb5d640 10 monclient(hunting): picked mon.noname-c con 0x7fe6780a98f0 addr v1:172.21.15.98:6789/0 2024-03-25T18:17:00.174 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.172+0000 7fe67cb5d640 10 monclient(hunting): picked mon.noname-a con 0x7fe67813f710 addr v1:172.21.15.97:6789/0 2024-03-25T18:17:00.174 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.172+0000 7fe67cb5d640 10 monclient(hunting): picked mon.noname-b con 0x7fe6781430c0 addr v1:172.21.15.97:6790/0 2024-03-25T18:17:00.174 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.172+0000 7fe67cb5d640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe678142520 con 0x7fe67813f710 2024-03-25T18:17:00.174 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.172+0000 7fe67cb5d640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6780b28a0 con 0x7fe6780a98f0 2024-03-25T18:17:00.174 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.172+0000 7fe67cb5d640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6780b2a30 con 0x7fe6781430c0 2024-03-25T18:17:00.175 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.172+0000 7fe67cb5d640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:00.175 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.172+0000 7fe67cb5d640 10 monclient(hunting): authenticate will time out at 7632.448864s 2024-03-25T18:17:00.187 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.185+0000 7fe675d9b640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fe67813f710 0x7fe6781429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:39732/0 (socket says 172.21.15.97:39732) 2024-03-25T18:17:00.187 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.185+0000 7fe675d9b640 1 -- 172.21.15.97:0/144296238 learned_addr learned my addr 172.21.15.97:0/144296238 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:17:00.196 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.195+0000 7fe67559a640 1 -- 172.21.15.97:0/144296238 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1415800180 0 0) 0x7fe678142520 con 0x7fe67813f710 2024-03-25T18:17:00.196 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.195+0000 7fe67559a640 10 monclient(hunting): _negotiate 2024-03-25T18:17:00.196 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.195+0000 7fe67559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:00.196 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.195+0000 7fe67559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:00.196 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.195+0000 7fe67559a640 10 monclient(hunting): my global_id is 4248 2024-03-25T18:17:00.196 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.195+0000 7fe67559a640 1 -- 172.21.15.97:0/144296238 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe660003520 con 0x7fe67813f710 2024-03-25T18:17:00.197 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.196+0000 7fe67559a640 1 -- 172.21.15.97:0/144296238 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4228467388 0 0) 0x7fe6780b2a30 con 0x7fe6781430c0 2024-03-25T18:17:00.197 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.196+0000 7fe67559a640 10 monclient(hunting): _negotiate 2024-03-25T18:17:00.197 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.196+0000 7fe67559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:00.197 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.196+0000 7fe67559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:00.197 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.196+0000 7fe67559a640 10 monclient(hunting): my global_id is 4241 2024-03-25T18:17:00.197 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.196+0000 7fe67559a640 1 -- 172.21.15.97:0/144296238 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe678142520 con 0x7fe6781430c0 2024-03-25T18:17:00.205 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.204+0000 7fe67559a640 1 -- 172.21.15.97:0/144296238 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1417250038 0 0) 0x7fe6780b28a0 con 0x7fe6780a98f0 2024-03-25T18:17:00.205 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.204+0000 7fe67559a640 10 monclient(hunting): _negotiate 2024-03-25T18:17:00.205 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.204+0000 7fe67559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:00.205 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.204+0000 7fe67559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:00.205 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.204+0000 7fe67559a640 10 monclient(hunting): my global_id is 4246 2024-03-25T18:17:00.206 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.204+0000 7fe67559a640 1 -- 172.21.15.97:0/144296238 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6780b2a30 con 0x7fe6780a98f0 2024-03-25T18:17:00.217 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.215+0000 7fe67559a640 1 -- 172.21.15.97:0/144296238 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2271722321 0 0) 0x7fe6780b2a30 con 0x7fe6780a98f0 2024-03-25T18:17:00.217 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.215+0000 7fe67559a640 1 -- 172.21.15.97:0/144296238 >> v1:172.21.15.97:6790/0 conn(0x7fe6781430c0 legacy=0x7fe67814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:00.217 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.215+0000 7fe67559a640 1 -- 172.21.15.97:0/144296238 >> v1:172.21.15.97:6789/0 conn(0x7fe67813f710 legacy=0x7fe6781429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:00.217 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.215+0000 7fe67559a640 10 monclient: _finish_hunting 0 2024-03-25T18:17:00.217 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.215+0000 7fe67559a640 1 monclient: found mon.noname-c 2024-03-25T18:17:00.217 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.215+0000 7fe67559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:00.217 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.215+0000 7fe67559a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.98:6789/0 2024-03-25T18:17:00.217 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.215+0000 7fe67559a640 1 -- 172.21.15.97:0/144296238 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6780b44c0 con 0x7fe6780a98f0 2024-03-25T18:17:00.217 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.215+0000 7fe67559a640 10 monclient: _finish_auth 0 2024-03-25T18:17:00.217 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.215+0000 7fe67559a640 10 monclient: _check_auth_tickets 2024-03-25T18:17:00.217 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.215+0000 7fe67559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:00.217 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.215+0000 7fe67559a640 1 -- 172.21.15.97:0/144296238 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7fe6680015b0 con 0x7fe6780a98f0 2024-03-25T18:17:00.217 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.215+0000 7fe67559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:00.217 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.215+0000 7fe67559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:17:00.218 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.215+0000 7fe67559a640 10 monclient: dump: 2024-03-25T18:17:00.218 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:17:00.218 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:17:00.218 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:17:00.218 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:17:00.218 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:00.218 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:17:00.218 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:17:00.218 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:17:00.218 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:17:00.218 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:17:00.218 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.215+0000 7fe67cb5d640 5 monclient: authenticate success, global_id 4246 2024-03-25T18:17:00.218 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.215+0000 7fe67cb5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:17:00.238 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.237+0000 7fe67559a640 1 -- 172.21.15.97:0/144296238 <== mon.1 v1:172.21.15.98:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe668001710 con 0x7fe6780a98f0 2024-03-25T18:17:00.238 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.237+0000 7fe67559a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:00.238 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.237+0000 7fe67559a640 1 -- 172.21.15.97:0/144296238 <== mon.1 v1:172.21.15.98:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7fe668001b60 con 0x7fe6780a98f0 2024-03-25T18:17:00.239 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.237+0000 7fe67cb5d640 10 monclient: get_monmap_and_config success 2024-03-25T18:17:00.239 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.237+0000 7fe67cb5d640 4 set_mon_vals no callback set 2024-03-25T18:17:00.239 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.237+0000 7fe67559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:00.239 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.237+0000 7fe67559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:17:00.239 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.237+0000 7fe67559a640 10 monclient: dump: 2024-03-25T18:17:00.239 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:17:00.239 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:17:00.239 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:17:00.239 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:17:00.239 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:00.239 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:17:00.239 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:17:00.239 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:17:00.239 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:17:00.240 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:17:00.241 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.238+0000 7fe67cb5d640 10 monclient: shutdown 2024-03-25T18:17:00.241 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.238+0000 7fe67cb5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:00.241 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.238+0000 7fe67cb5d640 1 -- 172.21.15.97:0/144296238 >> v1:172.21.15.98:6789/0 conn(0x7fe6780a98f0 legacy=0x7fe6780a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:00.241 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.238+0000 7fe67cb5d640 1 -- 172.21.15.97:0/144296238 shutdown_connections 2024-03-25T18:17:00.241 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.238+0000 7fe67cb5d640 1 -- 172.21.15.97:0/144296238 >> 172.21.15.97:0/144296238 conn(0x7fe6780b7ea0 msgr2=0x7fe6780b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:00.241 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.238+0000 7fe67cb5d640 1 -- 172.21.15.97:0/144296238 shutdown_connections 2024-03-25T18:17:00.241 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.238+0000 7fe67cb5d640 1 -- 172.21.15.97:0/144296238 wait complete. 2024-03-25T18:17:00.241 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.239+0000 7fe67cb5d640 10 monclient: build_initial_monmap 2024-03-25T18:17:00.242 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.239+0000 7fe67cb5d640 10 monclient: monmap: 2024-03-25T18:17:00.242 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:17:00.242 INFO:teuthology.misc.health.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:17:00.242 INFO:teuthology.misc.health.smithi097.stderr:last_changed 0.000000 2024-03-25T18:17:00.242 INFO:teuthology.misc.health.smithi097.stderr:created 0.000000 2024-03-25T18:17:00.242 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:00.242 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 1 2024-03-25T18:17:00.242 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:17:00.242 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T18:17:00.242 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-c 2024-03-25T18:17:00.242 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:17:00.242 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.239+0000 7fe67cb5d640 1 librados: starting msgr at 2024-03-25T18:17:00.242 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.240+0000 7fe67cb5d640 1 librados: starting objecter 2024-03-25T18:17:00.243 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.240+0000 7fe67cb5d640 1 Processor -- start 2024-03-25T18:17:00.243 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.240+0000 7fe67cb5d640 1 -- start start 2024-03-25T18:17:00.243 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.240+0000 7fe67cb5d640 1 librados: setting wanted keys 2024-03-25T18:17:00.243 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.240+0000 7fe67cb5d640 1 librados: calling monclient init 2024-03-25T18:17:00.243 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.240+0000 7fe67cb5d640 10 monclient: init 2024-03-25T18:17:00.243 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.240+0000 7fe67cb5d640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:00.244 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.240+0000 7fe67cb5d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:17:00.244 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.240+0000 7fe67cb5d640 10 monclient(hunting): picked mon.noname-b con 0x7fe6780a98f0 addr v1:172.21.15.98:6789/0 2024-03-25T18:17:00.244 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.240+0000 7fe67cb5d640 10 monclient(hunting): picked mon.noname-c con 0x7fe67813f710 addr v1:172.21.15.97:6790/0 2024-03-25T18:17:00.244 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.240+0000 7fe67cb5d640 10 monclient(hunting): picked mon.noname-a con 0x7fe6781430c0 addr v1:172.21.15.97:6789/0 2024-03-25T18:17:00.244 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.240+0000 7fe67cb5d640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6781db6c0 con 0x7fe6781430c0 2024-03-25T18:17:00.244 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.240+0000 7fe67cb5d640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6781daa90 con 0x7fe6780a98f0 2024-03-25T18:17:00.244 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.240+0000 7fe67cb5d640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe67814e8c0 con 0x7fe67813f710 2024-03-25T18:17:00.244 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.240+0000 7fe67cb5d640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:00.244 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.240+0000 7fe67cb5d640 10 monclient(hunting): authenticate will time out at 7632.516841s 2024-03-25T18:17:00.245 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.244+0000 7fe675d9b640 1 --1- >> v1:172.21.15.97:6790/0 conn(0x7fe67813f710 0x7fe6780b0de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.97:6790/0 says I am v1:172.21.15.97:55700/0 (socket says 172.21.15.97:55700) 2024-03-25T18:17:00.245 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.244+0000 7fe675d9b640 1 -- 172.21.15.97:0/1776069249 learned_addr learned my addr 172.21.15.97:0/1776069249 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:17:00.246 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.245+0000 7fe666ffd640 10 client.?.objecter ms_handle_connect 0x7fe67813f710 2024-03-25T18:17:00.246 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.245+0000 7fe666ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:00.252 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.251+0000 7fe666ffd640 10 client.?.objecter ms_handle_connect 0x7fe6781430c0 2024-03-25T18:17:00.252 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.251+0000 7fe666ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:00.260 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.259+0000 7fe666ffd640 1 -- 172.21.15.97:0/1776069249 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3800333557 0 0) 0x7fe67814e8c0 con 0x7fe67813f710 2024-03-25T18:17:00.260 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.259+0000 7fe666ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:17:00.260 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.259+0000 7fe666ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:00.260 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.259+0000 7fe666ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:00.260 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.259+0000 7fe666ffd640 10 monclient(hunting): my global_id is 4244 2024-03-25T18:17:00.260 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.259+0000 7fe666ffd640 1 -- 172.21.15.97:0/1776069249 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe658004eb0 con 0x7fe67813f710 2024-03-25T18:17:00.264 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.263+0000 7fe666ffd640 10 client.?.objecter ms_handle_connect 0x7fe6780a98f0 2024-03-25T18:17:00.264 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.263+0000 7fe666ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:00.273 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.272+0000 7fe666ffd640 1 -- 172.21.15.97:0/1776069249 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 800813733 0 0) 0x7fe6781db6c0 con 0x7fe6781430c0 2024-03-25T18:17:00.273 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.272+0000 7fe666ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:17:00.273 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.272+0000 7fe666ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:00.273 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.272+0000 7fe666ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:00.273 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.272+0000 7fe666ffd640 10 monclient(hunting): my global_id is 4251 2024-03-25T18:17:00.273 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.272+0000 7fe666ffd640 1 -- 172.21.15.97:0/1776069249 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe67814e8c0 con 0x7fe6781430c0 2024-03-25T18:17:00.281 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.280+0000 7fe666ffd640 1 -- 172.21.15.97:0/1776069249 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2483249691 0 0) 0x7fe6781daa90 con 0x7fe6780a98f0 2024-03-25T18:17:00.281 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.280+0000 7fe666ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:17:00.281 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.280+0000 7fe666ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:00.282 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.280+0000 7fe666ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:00.282 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.280+0000 7fe666ffd640 10 monclient(hunting): my global_id is 4249 2024-03-25T18:17:00.282 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.280+0000 7fe666ffd640 1 -- 172.21.15.97:0/1776069249 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6781db6c0 con 0x7fe6780a98f0 2024-03-25T18:17:00.284 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.283+0000 7fe666ffd640 1 -- 172.21.15.97:0/1776069249 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1982905054 0 0) 0x7fe67814e8c0 con 0x7fe6781430c0 2024-03-25T18:17:00.284 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.283+0000 7fe666ffd640 1 -- 172.21.15.97:0/1776069249 --> v1:172.21.15.97:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe6781daa90 con 0x7fe6781430c0 2024-03-25T18:17:00.285 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.283+0000 7fe666ffd640 1 -- 172.21.15.97:0/1776069249 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7fe670001b90 con 0x7fe6781430c0 2024-03-25T18:17:00.285 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.283+0000 7fe666ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:00.285 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.283+0000 7fe666ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:17:00.285 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.283+0000 7fe666ffd640 10 monclient(hunting): dump: 2024-03-25T18:17:00.285 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:17:00.285 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:17:00.285 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:17:00.285 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:17:00.285 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:00.285 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:17:00.285 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:17:00.285 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:17:00.285 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:17:00.285 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:17:00.286 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.283+0000 7fe666ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:17:00.297 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.296+0000 7fe666ffd640 1 -- 172.21.15.97:0/1776069249 <== mon.2 v1:172.21.15.97:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1936241775 0 0) 0x7fe658004eb0 con 0x7fe67813f710 2024-03-25T18:17:00.297 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.296+0000 7fe666ffd640 1 -- 172.21.15.97:0/1776069249 --> v1:172.21.15.97:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe67814e8c0 con 0x7fe67813f710 2024-03-25T18:17:00.297 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.296+0000 7fe666ffd640 1 -- 172.21.15.97:0/1776069249 <== mon.2 v1:172.21.15.97:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7fe66c001950 con 0x7fe67813f710 2024-03-25T18:17:00.297 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.296+0000 7fe666ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:00.298 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.296+0000 7fe666ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:17:00.298 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.296+0000 7fe666ffd640 10 monclient(hunting): dump: 2024-03-25T18:17:00.298 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:17:00.298 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:17:00.298 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:17:00.298 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:17:00.298 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:00.298 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:17:00.298 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:17:00.298 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:17:00.298 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:17:00.299 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:17:00.300 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.298+0000 7fe666ffd640 1 -- 172.21.15.97:0/1776069249 <== mon.0 v1:172.21.15.97:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 987205456 0 0) 0x7fe6781daa90 con 0x7fe6781430c0 2024-03-25T18:17:00.300 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.298+0000 7fe666ffd640 1 -- 172.21.15.97:0/1776069249 >> v1:172.21.15.97:6790/0 conn(0x7fe67813f710 legacy=0x7fe6780b0de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:00.300 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.298+0000 7fe666ffd640 1 -- 172.21.15.97:0/1776069249 >> v1:172.21.15.98:6789/0 conn(0x7fe6780a98f0 legacy=0x7fe6781dafb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:00.300 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.298+0000 7fe666ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:17:00.300 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.298+0000 7fe666ffd640 1 monclient: found mon.a 2024-03-25T18:17:00.300 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.298+0000 7fe666ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:00.300 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.298+0000 7fe666ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:17:00.300 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.298+0000 7fe666ffd640 1 -- 172.21.15.97:0/1776069249 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6780b14f0 con 0x7fe6781430c0 2024-03-25T18:17:00.300 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.298+0000 7fe666ffd640 10 monclient: _finish_auth 0 2024-03-25T18:17:00.300 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.298+0000 7fe666ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:17:00.300 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.298+0000 7fe666ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:00.300 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.298+0000 7fe67cb5d640 5 monclient: authenticate success, global_id 4251 2024-03-25T18:17:00.300 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.298+0000 7fe67cb5d640 10 monclient: _renew_subs 2024-03-25T18:17:00.300 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.298+0000 7fe67cb5d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:17:00.301 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.298+0000 7fe67cb5d640 1 -- 172.21.15.97:0/1776069249 --> v1:172.21.15.97:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe67814de20 con 0x7fe6781430c0 2024-03-25T18:17:00.301 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.298+0000 7fe67cb5d640 10 client.4251.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:17:00.301 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.298+0000 7fe67cb5d640 10 monclient: _renew_subs 2024-03-25T18:17:00.301 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.298+0000 7fe67cb5d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:17:00.301 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.298+0000 7fe67cb5d640 1 -- 172.21.15.97:0/1776069249 --> v1:172.21.15.97:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe67814e1c0 con 0x7fe6781430c0 2024-03-25T18:17:00.301 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.298+0000 7fe67cb5d640 1 librados: init done 2024-03-25T18:17:00.302 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.299+0000 7fe67cb5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:00.302 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.299+0000 7fe67cb5d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:00.302 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.299+0000 7fe67cb5d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:17:00.302 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.299+0000 7fe67cb5d640 1 -- 172.21.15.97:0/1776069249 --> v1:172.21.15.97:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6781da880 con 0x7fe6781430c0 2024-03-25T18:17:00.601 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.600+0000 7fe666ffd640 1 -- 172.21.15.97:0/1776069249 <== mon.0 v1:172.21.15.97:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe670002040 con 0x7fe6781430c0 2024-03-25T18:17:00.602 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.600+0000 7fe666ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:00.602 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.600+0000 7fe666ffd640 1 -- 172.21.15.97:0/1776069249 <== mon.0 v1:172.21.15.97:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7fe670003b30 con 0x7fe6781430c0 2024-03-25T18:17:00.602 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.600+0000 7fe666ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:00.602 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.600+0000 7fe666ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:17:00.602 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.600+0000 7fe666ffd640 10 monclient: dump: 2024-03-25T18:17:00.602 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:17:00.602 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:17:00.602 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:17:00.602 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:17:00.602 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:00.602 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:17:00.603 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:17:00.603 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:17:00.603 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:17:00.603 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:17:00.603 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.600+0000 7fe674d99640 4 set_mon_vals no callback set 2024-03-25T18:17:00.603 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.601+0000 7fe666ffd640 1 -- 172.21.15.97:0/1776069249 <== mon.0 v1:172.21.15.97:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2377622137 0 0) 0x7fe670019b00 con 0x7fe6781430c0 2024-03-25T18:17:00.603 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.601+0000 7fe67659c640 10 monclient: get_auth_request con 0x7fe65806df10 auth_method 0 2024-03-25T18:17:00.603 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.602+0000 7fe666ffd640 10 client.4251.objecter ms_handle_connect 0x7fe65806df10 2024-03-25T18:17:00.609 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.607+0000 7fe666ffd640 1 -- 172.21.15.97:0/1776069249 <== mon.0 v1:172.21.15.97:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 430711918 0 0) 0x7fe670053790 con 0x7fe6781430c0 2024-03-25T18:17:00.609 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.607+0000 7fe666ffd640 10 client.4251.objecter ms_dispatch 0x7fe678000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:17:00.609 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.607+0000 7fe666ffd640 3 client.4251.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:17:00.609 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.607+0000 7fe666ffd640 3 client.4251.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:17:00.609 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.607+0000 7fe666ffd640 20 client.4251.objecter dump_active .. 0 homeless 2024-03-25T18:17:00.609 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.607+0000 7fe666ffd640 1 -- 172.21.15.97:0/1776069249 <== mon.0 v1:172.21.15.97:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe67004f2a0 con 0x7fe6781430c0 2024-03-25T18:17:00.609 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.607+0000 7fe666ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:00.609 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.607+0000 7fe666ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:17:00.877 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.876+0000 7fe67cb5d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:00.878 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.876+0000 7fe67cb5d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:00.878 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.876+0000 7fe67cb5d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:17:00.878 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.876+0000 7fe67cb5d640 1 -- 172.21.15.97:0/1776069249 --> v1:172.21.15.97:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe6780627b0 con 0x7fe6781430c0 2024-03-25T18:17:00.925 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.924+0000 7fe666ffd640 1 -- 172.21.15.97:0/1776069249 <== mon.0 v1:172.21.15.97:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2169427412) 0x7fe6700511d0 con 0x7fe6781430c0 2024-03-25T18:17:00.925 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.924+0000 7fe666ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:00.925 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.924+0000 7fe666ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:17:00.926 INFO:teuthology.misc.health.smithi097.stdout: 2024-03-25T18:17:00.926 INFO:teuthology.misc.health.smithi097.stdout:{"epoch":10,"fsid":"4c3d472c-79b8-486b-9e3d-2738d4306754","created":"2024-03-25T18:16:14.737309+0000","modified":"2024-03-25T18:16:20.357492+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:20.357492+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cb5152b9-262f-45c3-94b8-18d0e81df558","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6daf8793-0838-473d-a7d6-9a0d2894c75d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"136f1493-3b9c-4f43-a2e7-641482d69dae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"397d538c-4f64-4daf-9e8b-662d09f91327","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4f1df08-663f-4bda-93f8-3878ab7c24b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c4b18d70-a3ce-4b15-9b70-eae5525bd761","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8dd4033a-de0e-4f8a-96dc-59f3c1f825b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0077722e-ffa6-4a3d-9891-929d9869b286","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:17:00.928 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.926+0000 7fe67cb5d640 10 librados: watch_flush enter 2024-03-25T18:17:00.928 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.926+0000 7fe67cb5d640 10 librados: watch_flush exit 2024-03-25T18:17:00.928 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.926+0000 7fe67cb5d640 20 client.4251.objecter shutdown clearing up homeless session... 2024-03-25T18:17:00.928 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.926+0000 7fe67cb5d640 10 client.4251.objecter successfully canceled tick 2024-03-25T18:17:00.928 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.926+0000 7fe67cb5d640 1 -- 172.21.15.97:0/1776069249 >> v1:172.21.15.98:6800/800387573 conn(0x7fe65806df10 legacy=0x7fe6580703d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:00.928 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.926+0000 7fe67cb5d640 10 monclient: shutdown 2024-03-25T18:17:00.928 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.926+0000 7fe67cb5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:00.928 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.926+0000 7fe67cb5d640 1 -- 172.21.15.97:0/1776069249 >> v1:172.21.15.97:6789/0 conn(0x7fe6781430c0 legacy=0x7fe6781e1bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:00.928 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.926+0000 7fe67cb5d640 1 -- 172.21.15.97:0/1776069249 shutdown_connections 2024-03-25T18:17:00.929 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.926+0000 7fe67cb5d640 1 -- 172.21.15.97:0/1776069249 >> 172.21.15.97:0/1776069249 conn(0x7fe6780b7ea0 msgr2=0x7fe678074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:00.929 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.926+0000 7fe67cb5d640 1 -- 172.21.15.97:0/1776069249 shutdown_connections 2024-03-25T18:17:00.929 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.926+0000 7fe67cb5d640 1 -- 172.21.15.97:0/1776069249 wait complete. 2024-03-25T18:17:00.929 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:00.926+0000 7fe67cb5d640 1 librados: shutdown 2024-03-25T18:17:00.940 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:17:07.746 DEBUG:teuthology.orchestra.run.smithi097:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:17:07.863 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:07.862+0000 7f0725158640 10 monclient: get_monmap_and_config 2024-03-25T18:17:07.864 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:07.862+0000 7f0725158640 10 monclient: build_initial_monmap 2024-03-25T18:17:07.864 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:07.862+0000 7f0725158640 10 monclient: monmap: 2024-03-25T18:17:07.864 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:17:07.864 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:17:07.864 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:17:07.864093+0000 2024-03-25T18:17:07.864 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:17:07.864093+0000 2024-03-25T18:17:07.864 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:07.864 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:17:07.864 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:17:07.864 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T18:17:07.864 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-b 2024-03-25T18:17:07.864 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:17:07.865 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:07.863+0000 7f0725158640 1 Processor -- start 2024-03-25T18:17:07.866 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:07.863+0000 7f0725158640 1 -- start start 2024-03-25T18:17:07.866 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:07.863+0000 7f0725158640 10 monclient: init 2024-03-25T18:17:07.866 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:07.864+0000 7f0725158640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:07.866 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:07.864+0000 7f0725158640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:17:07.866 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:07.864+0000 7f0725158640 10 monclient(hunting): picked mon.noname-b con 0x7f07200b4d60 addr v1:172.21.15.97:6790/0 2024-03-25T18:17:07.866 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:07.864+0000 7f0725158640 10 monclient(hunting): picked mon.noname-c con 0x7f07200b1730 addr v1:172.21.15.98:6789/0 2024-03-25T18:17:07.866 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:07.864+0000 7f0725158640 10 monclient(hunting): picked mon.noname-a con 0x7f072014be90 addr v1:172.21.15.97:6789/0 2024-03-25T18:17:07.866 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:07.864+0000 7f0725158640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07200af940 con 0x7f072014be90 2024-03-25T18:17:07.866 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:07.864+0000 7f0725158640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07200b0a70 con 0x7f07200b1730 2024-03-25T18:17:07.866 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:07.864+0000 7f0725158640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07200b0d50 con 0x7f07200b4d60 2024-03-25T18:17:07.866 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:07.864+0000 7f0725158640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:07.866 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:07.864+0000 7f0725158640 10 monclient(hunting): authenticate will time out at 7640.140268s 2024-03-25T18:17:07.871 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:07.870+0000 7f071e59c640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f07200b1730 0x7f07200b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.97:54796/0 (socket says 172.21.15.97:54796) 2024-03-25T18:17:07.871 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:07.870+0000 7f071e59c640 1 -- 172.21.15.97:0/3887192091 learned_addr learned my addr 172.21.15.97:0/3887192091 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:17:07.891 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:07.888+0000 7f071dd9b640 1 -- 172.21.15.97:0/3887192091 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2970504240 0 0) 0x7f07200af940 con 0x7f072014be90 2024-03-25T18:17:07.891 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:07.889+0000 7f071dd9b640 10 monclient(hunting): _negotiate 2024-03-25T18:17:07.892 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:07.889+0000 7f071dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:07.892 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:07.889+0000 7f071dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:07.892 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:07.889+0000 7f071dd9b640 10 monclient(hunting): my global_id is 4254 2024-03-25T18:17:07.892 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:07.889+0000 7f071dd9b640 1 -- 172.21.15.97:0/3887192091 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0708003520 con 0x7f072014be90 2024-03-25T18:17:07.892 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:07.889+0000 7f071dd9b640 1 -- 172.21.15.97:0/3887192091 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3870566257 0 0) 0x7f07200b0a70 con 0x7f07200b1730 2024-03-25T18:17:07.893 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:07.889+0000 7f071dd9b640 10 monclient(hunting): _negotiate 2024-03-25T18:17:07.893 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:07.889+0000 7f071dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:07.893 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:07.889+0000 7f071dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:07.893 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:07.889+0000 7f071dd9b640 10 monclient(hunting): my global_id is 4252 2024-03-25T18:17:07.893 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:07.889+0000 7f071dd9b640 1 -- 172.21.15.97:0/3887192091 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07200af940 con 0x7f07200b1730 2024-03-25T18:17:07.897 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:07.894+0000 7f071dd9b640 1 -- 172.21.15.97:0/3887192091 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1501672863 0 0) 0x7f07200b0d50 con 0x7f07200b4d60 2024-03-25T18:17:07.897 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:07.894+0000 7f071dd9b640 10 monclient(hunting): _negotiate 2024-03-25T18:17:07.898 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:07.894+0000 7f071dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:07.898 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:07.894+0000 7f071dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:07.898 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:07.894+0000 7f071dd9b640 10 monclient(hunting): my global_id is 4247 2024-03-25T18:17:07.898 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:07.894+0000 7f071dd9b640 1 -- 172.21.15.97:0/3887192091 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07200b0a70 con 0x7f07200b4d60 2024-03-25T18:17:07.903 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:07.902+0000 7f071dd9b640 1 -- 172.21.15.97:0/3887192091 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1436680925 0 0) 0x7f0708003520 con 0x7f072014be90 2024-03-25T18:17:07.904 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:07.902+0000 7f071dd9b640 1 -- 172.21.15.97:0/3887192091 >> v1:172.21.15.97:6790/0 conn(0x7f07200b4d60 legacy=0x7f072013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:07.904 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:07.902+0000 7f071dd9b640 1 -- 172.21.15.97:0/3887192091 >> v1:172.21.15.98:6789/0 conn(0x7f07200b1730 legacy=0x7f07200b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:07.904 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:07.902+0000 7f071dd9b640 10 monclient: _finish_hunting 0 2024-03-25T18:17:07.904 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:07.902+0000 7f071dd9b640 1 monclient: found mon.noname-a 2024-03-25T18:17:07.904 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:07.902+0000 7f071dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:07.904 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:07.902+0000 7f071dd9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.97:6789/0 2024-03-25T18:17:07.904 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:07.902+0000 7f071dd9b640 1 -- 172.21.15.97:0/3887192091 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07200b1030 con 0x7f072014be90 2024-03-25T18:17:07.904 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:07.902+0000 7f071dd9b640 10 monclient: _finish_auth 0 2024-03-25T18:17:07.904 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:07.902+0000 7f071dd9b640 10 monclient: _check_auth_tickets 2024-03-25T18:17:07.904 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:07.902+0000 7f071dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:07.904 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:07.902+0000 7f071dd9b640 1 -- 172.21.15.97:0/3887192091 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f0718001a10 con 0x7f072014be90 2024-03-25T18:17:07.904 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:07.902+0000 7f0725158640 5 monclient: authenticate success, global_id 4254 2024-03-25T18:17:07.904 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:07.902+0000 7f0725158640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:17:07.904 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:07.902+0000 7f071dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:07.905 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:07.902+0000 7f071dd9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:17:07.905 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:07.902+0000 7f071dd9b640 10 monclient: dump: 2024-03-25T18:17:07.905 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:17:07.905 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:17:07.905 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:17:07.905 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:17:07.905 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:07.905 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:17:07.905 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:17:07.905 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:17:07.905 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:17:07.905 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:17:07.905 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:07.902+0000 7f0725158640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:17:07.928 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:07.926+0000 7f071dd9b640 1 -- 172.21.15.97:0/3887192091 <== mon.0 v1:172.21.15.97:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0718001b70 con 0x7f072014be90 2024-03-25T18:17:07.928 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:07.926+0000 7f071dd9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:07.928 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:07.926+0000 7f071dd9b640 1 -- 172.21.15.97:0/3887192091 <== mon.0 v1:172.21.15.97:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f0718001f80 con 0x7f072014be90 2024-03-25T18:17:07.928 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:07.926+0000 7f0725158640 10 monclient: get_monmap_and_config success 2024-03-25T18:17:07.928 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:07.926+0000 7f0725158640 4 set_mon_vals no callback set 2024-03-25T18:17:07.928 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:07.926+0000 7f071dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:07.928 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:07.926+0000 7f071dd9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:17:07.928 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:07.926+0000 7f071dd9b640 10 monclient: dump: 2024-03-25T18:17:07.928 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:17:07.928 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:17:07.929 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:17:07.929 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:17:07.929 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:07.929 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:17:07.929 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:17:07.929 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:17:07.929 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:17:07.929 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:17:07.931 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:07.927+0000 7f0725158640 10 monclient: shutdown 2024-03-25T18:17:07.931 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:07.927+0000 7f0725158640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:07.931 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:07.927+0000 7f0725158640 1 -- 172.21.15.97:0/3887192091 >> v1:172.21.15.97:6789/0 conn(0x7f072014be90 legacy=0x7f072014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:07.931 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:07.927+0000 7f0725158640 1 -- 172.21.15.97:0/3887192091 shutdown_connections 2024-03-25T18:17:07.931 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:07.927+0000 7f0725158640 1 -- 172.21.15.97:0/3887192091 >> 172.21.15.97:0/3887192091 conn(0x7f07200617a0 msgr2=0x7f0720061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:07.931 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:07.927+0000 7f0725158640 1 -- 172.21.15.97:0/3887192091 shutdown_connections 2024-03-25T18:17:07.931 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:07.927+0000 7f0725158640 1 -- 172.21.15.97:0/3887192091 wait complete. 2024-03-25T18:17:07.932 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:07.928+0000 7f0725158640 10 monclient: build_initial_monmap 2024-03-25T18:17:07.932 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:07.928+0000 7f0725158640 10 monclient: monmap: 2024-03-25T18:17:07.932 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:17:07.932 INFO:teuthology.misc.health.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:17:07.932 INFO:teuthology.misc.health.smithi097.stderr:last_changed 0.000000 2024-03-25T18:17:07.932 INFO:teuthology.misc.health.smithi097.stderr:created 0.000000 2024-03-25T18:17:07.932 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:07.932 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 1 2024-03-25T18:17:07.932 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:17:07.932 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T18:17:07.932 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-c 2024-03-25T18:17:07.932 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:17:07.932 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:07.928+0000 7f0725158640 1 librados: starting msgr at 2024-03-25T18:17:07.933 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:07.928+0000 7f0725158640 1 librados: starting objecter 2024-03-25T18:17:07.933 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:07.929+0000 7f0725158640 1 Processor -- start 2024-03-25T18:17:07.933 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:07.929+0000 7f0725158640 1 -- start start 2024-03-25T18:17:07.933 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:07.929+0000 7f0725158640 1 librados: setting wanted keys 2024-03-25T18:17:07.933 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:07.929+0000 7f0725158640 1 librados: calling monclient init 2024-03-25T18:17:07.933 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:07.929+0000 7f0725158640 10 monclient: init 2024-03-25T18:17:07.933 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:07.929+0000 7f0725158640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:07.933 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:07.929+0000 7f0725158640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:17:07.933 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:07.929+0000 7f0725158640 10 monclient(hunting): picked mon.noname-b con 0x7f07200b1730 addr v1:172.21.15.98:6789/0 2024-03-25T18:17:07.933 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:07.929+0000 7f0725158640 10 monclient(hunting): picked mon.noname-a con 0x7f07200b4d60 addr v1:172.21.15.97:6789/0 2024-03-25T18:17:07.933 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:07.929+0000 7f0725158640 10 monclient(hunting): picked mon.noname-c con 0x7f072014be90 addr v1:172.21.15.97:6790/0 2024-03-25T18:17:07.933 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:07.929+0000 7f0725158640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07201d87c0 con 0x7f07200b4d60 2024-03-25T18:17:07.933 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:07.929+0000 7f0725158640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0720138820 con 0x7f07200b1730 2024-03-25T18:17:07.933 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:07.929+0000 7f0725158640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07201389b0 con 0x7f072014be90 2024-03-25T18:17:07.933 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:07.929+0000 7f0725158640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:07.934 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:07.929+0000 7f0725158640 10 monclient(hunting): authenticate will time out at 7640.205668s 2024-03-25T18:17:07.946 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:07.945+0000 7f071ed9d640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f07200b1730 0x7f0720138060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.97:54856/0 (socket says 172.21.15.97:54856) 2024-03-25T18:17:07.946 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:07.945+0000 7f071ed9d640 1 -- 172.21.15.97:0/91013321 learned_addr learned my addr 172.21.15.97:0/91013321 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:17:07.948 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:07.947+0000 7f07077fe640 10 client.?.objecter ms_handle_connect 0x7f07200b1730 2024-03-25T18:17:07.948 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:07.947+0000 7f07077fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:07.950 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:07.949+0000 7f07077fe640 10 client.?.objecter ms_handle_connect 0x7f07200b4d60 2024-03-25T18:17:07.950 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:07.949+0000 7f07077fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:07.959 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:07.958+0000 7f07077fe640 10 client.?.objecter ms_handle_connect 0x7f072014be90 2024-03-25T18:17:07.959 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:07.958+0000 7f07077fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:07.962 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:07.961+0000 7f07077fe640 1 -- 172.21.15.97:0/91013321 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1598484506 0 0) 0x7f07201d87c0 con 0x7f07200b4d60 2024-03-25T18:17:07.962 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:07.961+0000 7f07077fe640 10 monclient(hunting): _negotiate 2024-03-25T18:17:07.962 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:07.961+0000 7f07077fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:07.962 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:07.961+0000 7f07077fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:07.962 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:07.961+0000 7f07077fe640 10 monclient(hunting): my global_id is 4257 2024-03-25T18:17:07.962 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:07.961+0000 7f07077fe640 1 -- 172.21.15.97:0/91013321 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f070c004eb0 con 0x7f07200b4d60 2024-03-25T18:17:07.963 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:07.962+0000 7f07077fe640 1 -- 172.21.15.97:0/91013321 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1697278917 0 0) 0x7f0720138820 con 0x7f07200b1730 2024-03-25T18:17:07.963 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:07.962+0000 7f07077fe640 10 monclient(hunting): _negotiate 2024-03-25T18:17:07.963 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:07.962+0000 7f07077fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:07.963 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:07.962+0000 7f07077fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:07.963 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:07.962+0000 7f07077fe640 10 monclient(hunting): my global_id is 4255 2024-03-25T18:17:07.963 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:07.962+0000 7f07077fe640 1 -- 172.21.15.97:0/91013321 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07201d87c0 con 0x7f07200b1730 2024-03-25T18:17:07.980 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:07.979+0000 7f07077fe640 1 -- 172.21.15.97:0/91013321 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4204528746 0 0) 0x7f070c004eb0 con 0x7f07200b4d60 2024-03-25T18:17:07.980 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:07.979+0000 7f07077fe640 1 -- 172.21.15.97:0/91013321 --> v1:172.21.15.97:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0720138820 con 0x7f07200b4d60 2024-03-25T18:17:07.980 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:07.979+0000 7f07077fe640 1 -- 172.21.15.97:0/91013321 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f0714001970 con 0x7f07200b4d60 2024-03-25T18:17:07.980 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:07.979+0000 7f07077fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:07.980 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:07.979+0000 7f07077fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:17:07.980 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:07.979+0000 7f07077fe640 10 monclient(hunting): dump: 2024-03-25T18:17:07.980 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:17:07.980 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:17:07.981 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:17:07.981 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:17:07.981 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:07.981 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:17:07.981 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:17:07.981 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:17:07.981 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:17:07.981 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:17:07.981 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:07.979+0000 7f07077fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:17:07.983 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:07.982+0000 7f07077fe640 1 -- 172.21.15.97:0/91013321 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1140921910 0 0) 0x7f07201389b0 con 0x7f072014be90 2024-03-25T18:17:07.983 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:07.982+0000 7f07077fe640 10 monclient(hunting): _negotiate 2024-03-25T18:17:07.983 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:07.982+0000 7f07077fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:07.983 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:07.982+0000 7f07077fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:07.983 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:07.982+0000 7f07077fe640 10 monclient(hunting): my global_id is 4250 2024-03-25T18:17:07.983 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:07.982+0000 7f07077fe640 1 -- 172.21.15.97:0/91013321 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f070c004eb0 con 0x7f072014be90 2024-03-25T18:17:07.993 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:07.992+0000 7f07077fe640 1 -- 172.21.15.97:0/91013321 <== mon.2 v1:172.21.15.97:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1491089716 0 0) 0x7f070c004eb0 con 0x7f072014be90 2024-03-25T18:17:07.993 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:07.992+0000 7f07077fe640 1 -- 172.21.15.97:0/91013321 --> v1:172.21.15.97:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f07201389b0 con 0x7f072014be90 2024-03-25T18:17:07.998 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:07.997+0000 7f07077fe640 1 -- 172.21.15.97:0/91013321 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3601994797 0 0) 0x7f07201d87c0 con 0x7f07200b1730 2024-03-25T18:17:07.998 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:07.997+0000 7f07077fe640 1 -- 172.21.15.97:0/91013321 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f070c004eb0 con 0x7f07200b1730 2024-03-25T18:17:07.998 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:07.997+0000 7f07077fe640 1 -- 172.21.15.97:0/91013321 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f07100019f0 con 0x7f07200b1730 2024-03-25T18:17:07.998 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:07.997+0000 7f07077fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:07.998 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:07.997+0000 7f07077fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:17:07.998 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:07.997+0000 7f07077fe640 10 monclient(hunting): dump: 2024-03-25T18:17:07.998 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:17:07.998 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:17:07.998 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:17:07.998 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:17:07.999 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:07.999 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:17:07.999 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:17:07.999 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:17:07.999 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:17:07.999 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:17:08.001 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:08.000+0000 7f07077fe640 1 -- 172.21.15.97:0/91013321 <== mon.0 v1:172.21.15.97:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2493668409 0 0) 0x7f0720138820 con 0x7f07200b4d60 2024-03-25T18:17:08.001 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:08.000+0000 7f07077fe640 1 -- 172.21.15.97:0/91013321 >> v1:172.21.15.97:6790/0 conn(0x7f072014be90 legacy=0x7f07201e22a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:08.002 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:08.000+0000 7f07077fe640 1 -- 172.21.15.97:0/91013321 >> v1:172.21.15.98:6789/0 conn(0x7f07200b1730 legacy=0x7f0720138060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:08.002 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:08.000+0000 7f07077fe640 10 monclient: _finish_hunting 0 2024-03-25T18:17:08.002 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:08.000+0000 7f07077fe640 1 monclient: found mon.a 2024-03-25T18:17:08.002 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:08.000+0000 7f07077fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:08.002 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:08.000+0000 7f07077fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:17:08.002 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:08.000+0000 7f07077fe640 1 -- 172.21.15.97:0/91013321 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07201405f0 con 0x7f07200b4d60 2024-03-25T18:17:08.002 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:08.000+0000 7f07077fe640 10 monclient: _finish_auth 0 2024-03-25T18:17:08.002 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:08.000+0000 7f07077fe640 10 monclient: _check_auth_tickets 2024-03-25T18:17:08.002 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:08.000+0000 7f07077fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:08.002 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:08.000+0000 7f0725158640 5 monclient: authenticate success, global_id 4257 2024-03-25T18:17:08.002 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:08.000+0000 7f0725158640 10 monclient: _renew_subs 2024-03-25T18:17:08.002 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:08.000+0000 7f0725158640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:17:08.002 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:08.000+0000 7f0725158640 1 -- 172.21.15.97:0/91013321 --> v1:172.21.15.97:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07201407e0 con 0x7f07200b4d60 2024-03-25T18:17:08.002 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:08.000+0000 7f0725158640 10 client.4257.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:17:08.003 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:08.000+0000 7f0725158640 10 monclient: _renew_subs 2024-03-25T18:17:08.003 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:08.000+0000 7f0725158640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:17:08.003 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:08.000+0000 7f0725158640 1 -- 172.21.15.97:0/91013321 --> v1:172.21.15.97:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f07201e3ba0 con 0x7f07200b4d60 2024-03-25T18:17:08.003 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:08.000+0000 7f0725158640 1 librados: init done 2024-03-25T18:17:08.003 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:08.001+0000 7f0725158640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:08.003 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:08.001+0000 7f0725158640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:08.003 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:08.001+0000 7f0725158640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:17:08.004 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:08.001+0000 7f0725158640 1 -- 172.21.15.97:0/91013321 --> v1:172.21.15.97:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07200b5ac0 con 0x7f07200b4d60 2024-03-25T18:17:08.316 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:08.315+0000 7f07077fe640 1 -- 172.21.15.97:0/91013321 <== mon.0 v1:172.21.15.97:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f07140047e0 con 0x7f07200b4d60 2024-03-25T18:17:08.316 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:08.315+0000 7f07077fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:08.316 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:08.315+0000 7f07077fe640 1 -- 172.21.15.97:0/91013321 <== mon.0 v1:172.21.15.97:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f0714004af0 con 0x7f07200b4d60 2024-03-25T18:17:08.316 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:08.315+0000 7f07077fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:08.316 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:08.315+0000 7f07077fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:17:08.316 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:08.315+0000 7f07077fe640 10 monclient: dump: 2024-03-25T18:17:08.316 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:17:08.316 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:17:08.317 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:17:08.317 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:17:08.317 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:08.317 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:17:08.317 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:17:08.317 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:17:08.317 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:17:08.317 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:17:08.317 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:08.315+0000 7f071d59a640 4 set_mon_vals no callback set 2024-03-25T18:17:08.318 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:08.316+0000 7f07077fe640 1 -- 172.21.15.97:0/91013321 <== mon.0 v1:172.21.15.97:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2377622137 0 0) 0x7f071401ab40 con 0x7f07200b4d60 2024-03-25T18:17:08.318 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:08.316+0000 7f071ed9d640 10 monclient: get_auth_request con 0x7f070c06e2a0 auth_method 0 2024-03-25T18:17:08.318 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:08.316+0000 7f07077fe640 10 client.4257.objecter ms_handle_connect 0x7f070c06e2a0 2024-03-25T18:17:08.323 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:08.322+0000 7f07077fe640 1 -- 172.21.15.97:0/91013321 <== mon.0 v1:172.21.15.97:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 430711918 0 0) 0x7f07140547c0 con 0x7f07200b4d60 2024-03-25T18:17:08.323 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:08.322+0000 7f07077fe640 10 client.4257.objecter ms_dispatch 0x7f0720000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:17:08.323 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:08.322+0000 7f07077fe640 3 client.4257.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:17:08.323 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:08.322+0000 7f07077fe640 3 client.4257.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:17:08.323 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:08.322+0000 7f07077fe640 20 client.4257.objecter dump_active .. 0 homeless 2024-03-25T18:17:08.323 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:08.322+0000 7f07077fe640 1 -- 172.21.15.97:0/91013321 <== mon.0 v1:172.21.15.97:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0714050100 con 0x7f07200b4d60 2024-03-25T18:17:08.323 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:08.322+0000 7f07077fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:08.324 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:08.322+0000 7f07077fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:17:08.577 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:08.576+0000 7f0725158640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:08.577 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:08.576+0000 7f0725158640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:08.577 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:08.576+0000 7f0725158640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:17:08.577 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:08.576+0000 7f0725158640 1 -- 172.21.15.97:0/91013321 --> v1:172.21.15.97:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f07200627b0 con 0x7f07200b4d60 2024-03-25T18:17:08.642 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:08.640+0000 7f07077fe640 1 -- 172.21.15.97:0/91013321 <== mon.0 v1:172.21.15.97:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2169427412) 0x7f0714052200 con 0x7f07200b4d60 2024-03-25T18:17:08.642 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:08.641+0000 7f07077fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:08.642 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:08.641+0000 7f07077fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:17:08.642 INFO:teuthology.misc.health.smithi097.stdout: 2024-03-25T18:17:08.642 INFO:teuthology.misc.health.smithi097.stdout:{"epoch":10,"fsid":"4c3d472c-79b8-486b-9e3d-2738d4306754","created":"2024-03-25T18:16:14.737309+0000","modified":"2024-03-25T18:16:20.357492+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:20.357492+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cb5152b9-262f-45c3-94b8-18d0e81df558","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6daf8793-0838-473d-a7d6-9a0d2894c75d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"136f1493-3b9c-4f43-a2e7-641482d69dae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"397d538c-4f64-4daf-9e8b-662d09f91327","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4f1df08-663f-4bda-93f8-3878ab7c24b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c4b18d70-a3ce-4b15-9b70-eae5525bd761","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8dd4033a-de0e-4f8a-96dc-59f3c1f825b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0077722e-ffa6-4a3d-9891-929d9869b286","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:17:08.646 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:08.644+0000 7f0725158640 10 librados: watch_flush enter 2024-03-25T18:17:08.646 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:08.644+0000 7f0725158640 10 librados: watch_flush exit 2024-03-25T18:17:08.646 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:08.644+0000 7f0725158640 20 client.4257.objecter shutdown clearing up homeless session... 2024-03-25T18:17:08.646 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:08.644+0000 7f0725158640 10 client.4257.objecter successfully canceled tick 2024-03-25T18:17:08.646 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:08.644+0000 7f0725158640 1 -- 172.21.15.97:0/91013321 >> v1:172.21.15.98:6800/800387573 conn(0x7f070c06e2a0 legacy=0x7f070c070760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:08.646 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:08.644+0000 7f0725158640 10 monclient: shutdown 2024-03-25T18:17:08.646 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:08.644+0000 7f0725158640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:08.646 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:08.644+0000 7f0725158640 1 -- 172.21.15.97:0/91013321 >> v1:172.21.15.97:6789/0 conn(0x7f07200b4d60 legacy=0x7f07201d80b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:08.646 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:08.644+0000 7f0725158640 1 -- 172.21.15.97:0/91013321 shutdown_connections 2024-03-25T18:17:08.646 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:08.644+0000 7f0725158640 1 -- 172.21.15.97:0/91013321 >> 172.21.15.97:0/91013321 conn(0x7f07200617a0 msgr2=0x7f07200b84e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:08.647 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:08.644+0000 7f0725158640 1 -- 172.21.15.97:0/91013321 shutdown_connections 2024-03-25T18:17:08.647 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:08.644+0000 7f0725158640 1 -- 172.21.15.97:0/91013321 wait complete. 2024-03-25T18:17:08.647 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:08.644+0000 7f0725158640 1 librados: shutdown 2024-03-25T18:17:08.657 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:17:11.944 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:01:13.983 34931== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:12.007 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:01:14.045 34931== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:12.011 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:01:14.054 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:12.069 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:01:14.113 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:12.144 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:01:14.183 34935== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:12.208 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:01:14.247 34935== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:12.234 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:01:14.272 34931== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:12.292 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:01:14.336 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:12.441 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:01:14.480 34935== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:15.464 DEBUG:teuthology.orchestra.run.smithi097:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:17:15.577 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.576+0000 7f5cfb627640 10 monclient: get_monmap_and_config 2024-03-25T18:17:15.578 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.576+0000 7f5cfb627640 10 monclient: build_initial_monmap 2024-03-25T18:17:15.578 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.576+0000 7f5cfb627640 10 monclient: monmap: 2024-03-25T18:17:15.578 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:17:15.578 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:17:15.578 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:17:15.577862+0000 2024-03-25T18:17:15.578 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:17:15.577862+0000 2024-03-25T18:17:15.578 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:15.578 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:17:15.578 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:17:15.578 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T18:17:15.578 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-b 2024-03-25T18:17:15.578 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:17:15.580 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.577+0000 7f5cfb627640 1 Processor -- start 2024-03-25T18:17:15.580 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.577+0000 7f5cfb627640 1 -- start start 2024-03-25T18:17:15.580 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.577+0000 7f5cfb627640 10 monclient: init 2024-03-25T18:17:15.580 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.578+0000 7f5cfb627640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:15.580 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.578+0000 7f5cfb627640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:17:15.580 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.578+0000 7f5cfb627640 10 monclient(hunting): picked mon.noname-a con 0x7f5cf40a98f0 addr v1:172.21.15.97:6789/0 2024-03-25T18:17:15.580 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.578+0000 7f5cfb627640 10 monclient(hunting): picked mon.noname-c con 0x7f5cf413f710 addr v1:172.21.15.98:6789/0 2024-03-25T18:17:15.580 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.578+0000 7f5cfb627640 10 monclient(hunting): picked mon.noname-b con 0x7f5cf41430c0 addr v1:172.21.15.97:6790/0 2024-03-25T18:17:15.580 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.578+0000 7f5cfb627640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5cf4142520 con 0x7f5cf40a98f0 2024-03-25T18:17:15.580 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.578+0000 7f5cfb627640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5cf40b28a0 con 0x7f5cf413f710 2024-03-25T18:17:15.580 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.578+0000 7f5cfb627640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5cf40b2a30 con 0x7f5cf41430c0 2024-03-25T18:17:15.580 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.578+0000 7f5cfb627640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:15.581 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.578+0000 7f5cfb627640 10 monclient(hunting): authenticate will time out at 7647.854254s 2024-03-25T18:17:15.588 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.587+0000 7f5cf9bc4640 1 --1- >> v1:172.21.15.97:6790/0 conn(0x7f5cf41430c0 0x7f5cf414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6790/0 says I am v1:172.21.15.97:48160/0 (socket says 172.21.15.97:48160) 2024-03-25T18:17:15.589 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.587+0000 7f5cf9bc4640 1 -- 172.21.15.97:0/2823317220 learned_addr learned my addr 172.21.15.97:0/2823317220 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:17:15.599 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.598+0000 7f5cebfff640 1 -- 172.21.15.97:0/2823317220 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1800664434 0 0) 0x7f5cf40b2a30 con 0x7f5cf41430c0 2024-03-25T18:17:15.599 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.598+0000 7f5cebfff640 10 monclient(hunting): _negotiate 2024-03-25T18:17:15.599 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.598+0000 7f5cebfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:15.599 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.598+0000 7f5cebfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:15.599 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.598+0000 7f5cebfff640 10 monclient(hunting): my global_id is 4253 2024-03-25T18:17:15.600 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.598+0000 7f5cebfff640 1 -- 172.21.15.97:0/2823317220 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ce0003520 con 0x7f5cf41430c0 2024-03-25T18:17:15.609 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.608+0000 7f5cebfff640 1 -- 172.21.15.97:0/2823317220 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3504154149 0 0) 0x7f5cf4142520 con 0x7f5cf40a98f0 2024-03-25T18:17:15.610 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.608+0000 7f5cebfff640 10 monclient(hunting): _negotiate 2024-03-25T18:17:15.610 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.608+0000 7f5cebfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:15.610 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.608+0000 7f5cebfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:15.610 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.608+0000 7f5cebfff640 10 monclient(hunting): my global_id is 4260 2024-03-25T18:17:15.610 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.608+0000 7f5cebfff640 1 -- 172.21.15.97:0/2823317220 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5cf40b2a30 con 0x7f5cf40a98f0 2024-03-25T18:17:15.691 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.689+0000 7f5cebfff640 1 -- 172.21.15.97:0/2823317220 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2021648489 0 0) 0x7f5cf40b2a30 con 0x7f5cf40a98f0 2024-03-25T18:17:15.691 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.689+0000 7f5cebfff640 1 -- 172.21.15.97:0/2823317220 >> v1:172.21.15.97:6790/0 conn(0x7f5cf41430c0 legacy=0x7f5cf414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:15.691 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.689+0000 7f5cebfff640 1 -- 172.21.15.97:0/2823317220 >> v1:172.21.15.98:6789/0 conn(0x7f5cf413f710 legacy=0x7f5cf41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:15.691 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.689+0000 7f5cebfff640 10 monclient: _finish_hunting 0 2024-03-25T18:17:15.691 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.689+0000 7f5cebfff640 1 monclient: found mon.noname-a 2024-03-25T18:17:15.691 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.689+0000 7f5cebfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:15.691 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.689+0000 7f5cebfff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.97:6789/0 2024-03-25T18:17:15.691 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.689+0000 7f5cebfff640 1 -- 172.21.15.97:0/2823317220 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cf40b44c0 con 0x7f5cf40a98f0 2024-03-25T18:17:15.691 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.689+0000 7f5cebfff640 10 monclient: _finish_auth 0 2024-03-25T18:17:15.691 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.689+0000 7f5cebfff640 10 monclient: _check_auth_tickets 2024-03-25T18:17:15.691 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.689+0000 7f5cebfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:15.691 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.689+0000 7f5cebfff640 1 -- 172.21.15.97:0/2823317220 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f5cf0001670 con 0x7f5cf40a98f0 2024-03-25T18:17:15.692 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.689+0000 7f5cfb627640 5 monclient: authenticate success, global_id 4260 2024-03-25T18:17:15.692 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.689+0000 7f5cfb627640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:17:15.692 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.689+0000 7f5cebfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:15.692 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.689+0000 7f5cebfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:17:15.692 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.689+0000 7f5cebfff640 10 monclient: dump: 2024-03-25T18:17:15.692 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:17:15.692 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:17:15.692 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:17:15.692 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:17:15.692 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:15.692 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:17:15.692 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:17:15.692 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:17:15.692 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:17:15.692 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:17:15.693 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.689+0000 7f5cfb627640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:17:15.700 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.699+0000 7f5cebfff640 1 -- 172.21.15.97:0/2823317220 <== mon.0 v1:172.21.15.97:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5cf00017d0 con 0x7f5cf40a98f0 2024-03-25T18:17:15.700 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.699+0000 7f5cebfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:15.700 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.699+0000 7f5cebfff640 1 -- 172.21.15.97:0/2823317220 <== mon.0 v1:172.21.15.97:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f5cf0001c20 con 0x7f5cf40a98f0 2024-03-25T18:17:15.701 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.699+0000 7f5cebfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:15.701 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.699+0000 7f5cebfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:17:15.701 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.699+0000 7f5cebfff640 10 monclient: dump: 2024-03-25T18:17:15.701 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:17:15.701 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:17:15.701 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:17:15.701 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:17:15.701 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:15.701 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:17:15.701 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:17:15.701 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:17:15.701 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:17:15.701 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:17:15.701 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.699+0000 7f5cfb627640 10 monclient: get_monmap_and_config success 2024-03-25T18:17:15.701 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.699+0000 7f5cfb627640 4 set_mon_vals no callback set 2024-03-25T18:17:15.702 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.700+0000 7f5cfb627640 10 monclient: shutdown 2024-03-25T18:17:15.703 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.700+0000 7f5cfb627640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:15.703 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.700+0000 7f5cfb627640 1 -- 172.21.15.97:0/2823317220 >> v1:172.21.15.97:6789/0 conn(0x7f5cf40a98f0 legacy=0x7f5cf40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:15.703 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.700+0000 7f5cfb627640 1 -- 172.21.15.97:0/2823317220 shutdown_connections 2024-03-25T18:17:15.703 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.700+0000 7f5cfb627640 1 -- 172.21.15.97:0/2823317220 >> 172.21.15.97:0/2823317220 conn(0x7f5cf40b7ea0 msgr2=0x7f5cf40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:15.703 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.700+0000 7f5cfb627640 1 -- 172.21.15.97:0/2823317220 shutdown_connections 2024-03-25T18:17:15.703 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.700+0000 7f5cfb627640 1 -- 172.21.15.97:0/2823317220 wait complete. 2024-03-25T18:17:15.703 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.701+0000 7f5cfb627640 10 monclient: build_initial_monmap 2024-03-25T18:17:15.703 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.701+0000 7f5cfb627640 10 monclient: monmap: 2024-03-25T18:17:15.703 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:17:15.703 INFO:teuthology.misc.health.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:17:15.703 INFO:teuthology.misc.health.smithi097.stderr:last_changed 0.000000 2024-03-25T18:17:15.703 INFO:teuthology.misc.health.smithi097.stderr:created 0.000000 2024-03-25T18:17:15.703 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:15.703 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 1 2024-03-25T18:17:15.704 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:17:15.704 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T18:17:15.704 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-c 2024-03-25T18:17:15.704 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:17:15.705 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.702+0000 7f5cfb627640 1 librados: starting msgr at 2024-03-25T18:17:15.705 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.702+0000 7f5cfb627640 1 librados: starting objecter 2024-03-25T18:17:15.705 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.702+0000 7f5cfb627640 1 Processor -- start 2024-03-25T18:17:15.705 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.702+0000 7f5cfb627640 1 -- start start 2024-03-25T18:17:15.705 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.702+0000 7f5cfb627640 1 librados: setting wanted keys 2024-03-25T18:17:15.705 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.702+0000 7f5cfb627640 1 librados: calling monclient init 2024-03-25T18:17:15.705 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.702+0000 7f5cfb627640 10 monclient: init 2024-03-25T18:17:15.705 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.702+0000 7f5cfb627640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:15.705 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.702+0000 7f5cfb627640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:17:15.705 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.702+0000 7f5cfb627640 10 monclient(hunting): picked mon.noname-a con 0x7f5cf40a98f0 addr v1:172.21.15.97:6789/0 2024-03-25T18:17:15.705 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.702+0000 7f5cfb627640 10 monclient(hunting): picked mon.noname-c con 0x7f5cf413f710 addr v1:172.21.15.97:6790/0 2024-03-25T18:17:15.706 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.702+0000 7f5cfb627640 10 monclient(hunting): picked mon.noname-b con 0x7f5cf41430c0 addr v1:172.21.15.98:6789/0 2024-03-25T18:17:15.706 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.702+0000 7f5cfb627640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5cf41daa20 con 0x7f5cf40a98f0 2024-03-25T18:17:15.706 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.703+0000 7f5cfb627640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5cf41371e0 con 0x7f5cf41430c0 2024-03-25T18:17:15.706 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.703+0000 7f5cfb627640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5cf41374c0 con 0x7f5cf413f710 2024-03-25T18:17:15.706 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.703+0000 7f5cfb627640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:15.706 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.703+0000 7f5cfb627640 10 monclient(hunting): authenticate will time out at 7647.978894s 2024-03-25T18:17:15.711 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.710+0000 7f5cf93c3640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f5cf40a98f0 0x7f5cf41352e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:58178/0 (socket says 172.21.15.97:58178) 2024-03-25T18:17:15.711 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.710+0000 7f5cf93c3640 1 -- 172.21.15.97:0/259485361 learned_addr learned my addr 172.21.15.97:0/259485361 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:17:15.712 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.711+0000 7f5ce9ffb640 10 client.?.objecter ms_handle_connect 0x7f5cf40a98f0 2024-03-25T18:17:15.712 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.711+0000 7f5ce9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:15.727 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.726+0000 7f5ce9ffb640 1 -- 172.21.15.97:0/259485361 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2666197787 0 0) 0x7f5cf41daa20 con 0x7f5cf40a98f0 2024-03-25T18:17:15.727 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.726+0000 7f5ce9ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:17:15.727 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.726+0000 7f5ce9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:15.727 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.726+0000 7f5ce9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:15.727 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.726+0000 7f5ce9ffb640 10 monclient(hunting): my global_id is 4263 2024-03-25T18:17:15.727 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.726+0000 7f5ce9ffb640 1 -- 172.21.15.97:0/259485361 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5cd8004eb0 con 0x7f5cf40a98f0 2024-03-25T18:17:15.745 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.744+0000 7f5ce9ffb640 10 client.?.objecter ms_handle_connect 0x7f5cf41430c0 2024-03-25T18:17:15.746 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.744+0000 7f5ce9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:15.749 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.747+0000 7f5ce9ffb640 1 -- 172.21.15.97:0/259485361 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2807451309 0 0) 0x7f5cf41371e0 con 0x7f5cf41430c0 2024-03-25T18:17:15.749 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.747+0000 7f5ce9ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:17:15.749 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.747+0000 7f5ce9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:15.749 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.747+0000 7f5ce9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:15.749 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.747+0000 7f5ce9ffb640 10 monclient(hunting): my global_id is 4261 2024-03-25T18:17:15.749 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.747+0000 7f5ce9ffb640 1 -- 172.21.15.97:0/259485361 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5cf41daa20 con 0x7f5cf41430c0 2024-03-25T18:17:15.749 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.747+0000 7f5ce9ffb640 1 -- 172.21.15.97:0/259485361 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4065566549 0 0) 0x7f5cd8004eb0 con 0x7f5cf40a98f0 2024-03-25T18:17:15.749 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.748+0000 7f5ce9ffb640 1 -- 172.21.15.97:0/259485361 --> v1:172.21.15.97:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5cf41371e0 con 0x7f5cf40a98f0 2024-03-25T18:17:15.750 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.748+0000 7f5ce9ffb640 1 -- 172.21.15.97:0/259485361 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f5cf0001630 con 0x7f5cf40a98f0 2024-03-25T18:17:15.750 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.748+0000 7f5ce9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:15.750 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.748+0000 7f5ce9ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:17:15.750 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.748+0000 7f5ce9ffb640 10 monclient(hunting): dump: 2024-03-25T18:17:15.750 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:17:15.750 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:17:15.750 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:17:15.750 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:17:15.750 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:15.750 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:17:15.750 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:17:15.750 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:17:15.751 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:17:15.751 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:17:15.751 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.748+0000 7f5ce9ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:17:15.751 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.748+0000 7f5ce9ffb640 10 client.?.objecter ms_handle_connect 0x7f5cf413f710 2024-03-25T18:17:15.751 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.748+0000 7f5ce9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:15.758 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.757+0000 7f5ce9ffb640 1 -- 172.21.15.97:0/259485361 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3133603728 0 0) 0x7f5cf41daa20 con 0x7f5cf41430c0 2024-03-25T18:17:15.758 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.757+0000 7f5ce9ffb640 1 -- 172.21.15.97:0/259485361 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5cd8004eb0 con 0x7f5cf41430c0 2024-03-25T18:17:15.758 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.757+0000 7f5ce9ffb640 1 -- 172.21.15.97:0/259485361 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f5cec001c10 con 0x7f5cf41430c0 2024-03-25T18:17:15.758 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.757+0000 7f5ce9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:15.758 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.757+0000 7f5ce9ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:17:15.758 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.757+0000 7f5ce9ffb640 10 monclient(hunting): dump: 2024-03-25T18:17:15.758 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:17:15.758 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:17:15.758 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:17:15.758 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:17:15.759 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:15.759 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:17:15.759 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:17:15.759 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:17:15.759 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:17:15.759 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:17:15.764 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.763+0000 7f5ce9ffb640 1 -- 172.21.15.97:0/259485361 <== mon.1 v1:172.21.15.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3283989302 0 0) 0x7f5cd8004eb0 con 0x7f5cf41430c0 2024-03-25T18:17:15.764 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.763+0000 7f5ce9ffb640 1 -- 172.21.15.97:0/259485361 >> v1:172.21.15.97:6790/0 conn(0x7f5cf413f710 legacy=0x7f5cf41daf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:15.765 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.763+0000 7f5ce9ffb640 1 -- 172.21.15.97:0/259485361 >> v1:172.21.15.97:6789/0 conn(0x7f5cf40a98f0 legacy=0x7f5cf41352e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:15.765 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.763+0000 7f5ce9ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:17:15.765 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.763+0000 7f5ce9ffb640 1 monclient: found mon.b 2024-03-25T18:17:15.765 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.763+0000 7f5ce9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:15.765 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.763+0000 7f5ce9ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:17:15.765 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.763+0000 7f5ce9ffb640 1 -- 172.21.15.97:0/259485361 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cf413c040 con 0x7f5cf41430c0 2024-03-25T18:17:15.765 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.763+0000 7f5ce9ffb640 10 monclient: _finish_auth 0 2024-03-25T18:17:15.765 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.763+0000 7f5ce9ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:17:15.765 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.763+0000 7f5ce9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:15.765 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.763+0000 7f5cfb627640 5 monclient: authenticate success, global_id 4261 2024-03-25T18:17:15.765 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.763+0000 7f5cfb627640 10 monclient: _renew_subs 2024-03-25T18:17:15.765 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.763+0000 7f5cfb627640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:17:15.765 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.763+0000 7f5cfb627640 1 -- 172.21.15.97:0/259485361 --> v1:172.21.15.98:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5cf41359f0 con 0x7f5cf41430c0 2024-03-25T18:17:15.766 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.763+0000 7f5cfb627640 10 client.4261.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:17:15.766 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.763+0000 7f5cfb627640 10 monclient: _renew_subs 2024-03-25T18:17:15.766 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.763+0000 7f5cfb627640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:17:15.766 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.763+0000 7f5cfb627640 1 -- 172.21.15.97:0/259485361 --> v1:172.21.15.98:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5cf41d75d0 con 0x7f5cf41430c0 2024-03-25T18:17:15.766 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.763+0000 7f5cfb627640 1 librados: init done 2024-03-25T18:17:15.767 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.764+0000 7f5cfb627640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:15.767 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.764+0000 7f5cfb627640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:15.767 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.764+0000 7f5cfb627640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:17:15.767 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.765+0000 7f5cfb627640 1 -- 172.21.15.97:0/259485361 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5cf40627b0 con 0x7f5cf41430c0 2024-03-25T18:17:15.782 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.781+0000 7f5ce9ffb640 1 -- 172.21.15.97:0/259485361 <== mon.1 v1:172.21.15.98:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5cec002020 con 0x7f5cf41430c0 2024-03-25T18:17:15.782 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.781+0000 7f5ce9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:15.782 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.781+0000 7f5ce9ffb640 1 -- 172.21.15.97:0/259485361 <== mon.1 v1:172.21.15.98:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f5cec004d00 con 0x7f5cf41430c0 2024-03-25T18:17:15.782 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.781+0000 7f5ce9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:15.782 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.781+0000 7f5ce9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:17:15.782 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.781+0000 7f5ce9ffb640 10 monclient: dump: 2024-03-25T18:17:15.782 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:17:15.782 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:17:15.782 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:17:15.783 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:17:15.783 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:15.783 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:17:15.783 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:17:15.783 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:17:15.783 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:17:15.783 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:17:15.783 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.781+0000 7f5ceb7fe640 4 set_mon_vals no callback set 2024-03-25T18:17:15.784 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.782+0000 7f5ce9ffb640 1 -- 172.21.15.97:0/259485361 <== mon.1 v1:172.21.15.98:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2377622137 0 0) 0x7f5cec004f20 con 0x7f5cf41430c0 2024-03-25T18:17:15.784 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.782+0000 7f5cf93c3640 10 monclient: get_auth_request con 0x7f5cd806ded0 auth_method 0 2024-03-25T18:17:15.784 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:15.783+0000 7f5ce9ffb640 10 client.4261.objecter ms_handle_connect 0x7f5cd806ded0 2024-03-25T18:17:16.041 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:16.040+0000 7f5ce9ffb640 1 -- 172.21.15.97:0/259485361 <== mon.1 v1:172.21.15.98:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 430711918 0 0) 0x7f5cec054990 con 0x7f5cf41430c0 2024-03-25T18:17:16.041 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:16.040+0000 7f5ce9ffb640 10 client.4261.objecter ms_dispatch 0x7f5cf4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:17:16.041 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:16.040+0000 7f5ce9ffb640 3 client.4261.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:17:16.042 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:16.040+0000 7f5ce9ffb640 3 client.4261.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:17:16.042 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:16.040+0000 7f5ce9ffb640 20 client.4261.objecter dump_active .. 0 homeless 2024-03-25T18:17:16.042 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:16.040+0000 7f5ce9ffb640 1 -- 172.21.15.97:0/259485361 <== mon.1 v1:172.21.15.98:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5cec0504a0 con 0x7f5cf41430c0 2024-03-25T18:17:16.042 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:16.040+0000 7f5ce9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:16.042 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:16.040+0000 7f5ce9ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:17:16.297 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:16.296+0000 7f5cfb627640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:16.298 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:16.296+0000 7f5cfb627640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:16.298 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:16.296+0000 7f5cfb627640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:17:16.298 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:16.296+0000 7f5cfb627640 1 -- 172.21.15.97:0/259485361 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5cf40008d0 con 0x7f5cf41430c0 2024-03-25T18:17:16.318 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:16.317+0000 7f5ce9ffb640 1 -- 172.21.15.97:0/259485361 <== mon.1 v1:172.21.15.98:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2169427412) 0x7f5cec0523d0 con 0x7f5cf41430c0 2024-03-25T18:17:16.318 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:16.317+0000 7f5ce9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:16.318 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:16.317+0000 7f5ce9ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:17:16.318 INFO:teuthology.misc.health.smithi097.stdout: 2024-03-25T18:17:16.319 INFO:teuthology.misc.health.smithi097.stdout:{"epoch":10,"fsid":"4c3d472c-79b8-486b-9e3d-2738d4306754","created":"2024-03-25T18:16:14.737309+0000","modified":"2024-03-25T18:16:20.357492+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:20.357492+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cb5152b9-262f-45c3-94b8-18d0e81df558","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6daf8793-0838-473d-a7d6-9a0d2894c75d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"136f1493-3b9c-4f43-a2e7-641482d69dae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"397d538c-4f64-4daf-9e8b-662d09f91327","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4f1df08-663f-4bda-93f8-3878ab7c24b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c4b18d70-a3ce-4b15-9b70-eae5525bd761","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8dd4033a-de0e-4f8a-96dc-59f3c1f825b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0077722e-ffa6-4a3d-9891-929d9869b286","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:17:16.321 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:16.319+0000 7f5cfb627640 10 librados: watch_flush enter 2024-03-25T18:17:16.321 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:16.319+0000 7f5cfb627640 10 librados: watch_flush exit 2024-03-25T18:17:16.321 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:16.319+0000 7f5cfb627640 20 client.4261.objecter shutdown clearing up homeless session... 2024-03-25T18:17:16.321 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:16.319+0000 7f5cfb627640 10 client.4261.objecter successfully canceled tick 2024-03-25T18:17:16.321 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:16.319+0000 7f5cfb627640 1 -- 172.21.15.97:0/259485361 >> v1:172.21.15.98:6800/800387573 conn(0x7f5cd806ded0 legacy=0x7f5cd8070390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:16.321 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:16.319+0000 7f5cfb627640 10 monclient: shutdown 2024-03-25T18:17:16.321 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:16.319+0000 7f5cfb627640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:16.321 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:16.319+0000 7f5cfb627640 1 -- 172.21.15.97:0/259485361 >> v1:172.21.15.98:6789/0 conn(0x7f5cf41430c0 legacy=0x7f5cf41e1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:16.321 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:16.319+0000 7f5cfb627640 1 -- 172.21.15.97:0/259485361 shutdown_connections 2024-03-25T18:17:16.321 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:16.319+0000 7f5cfb627640 1 -- 172.21.15.97:0/259485361 >> 172.21.15.97:0/259485361 conn(0x7f5cf40b7ea0 msgr2=0x7f5cf4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:16.321 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:16.319+0000 7f5cfb627640 1 -- 172.21.15.97:0/259485361 shutdown_connections 2024-03-25T18:17:16.322 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:16.319+0000 7f5cfb627640 1 -- 172.21.15.97:0/259485361 wait complete. 2024-03-25T18:17:16.322 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:16.319+0000 7f5cfb627640 1 librados: shutdown 2024-03-25T18:17:16.333 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:17:23.139 DEBUG:teuthology.orchestra.run.smithi097:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:17:23.241 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.240+0000 7f33d70a5640 10 monclient: get_monmap_and_config 2024-03-25T18:17:23.242 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.240+0000 7f33d70a5640 10 monclient: build_initial_monmap 2024-03-25T18:17:23.242 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.240+0000 7f33d70a5640 10 monclient: monmap: 2024-03-25T18:17:23.242 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:17:23.242 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:17:23.242 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:17:23.242100+0000 2024-03-25T18:17:23.242 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:17:23.242100+0000 2024-03-25T18:17:23.242 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:23.242 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:17:23.242 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:17:23.242 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T18:17:23.242 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-b 2024-03-25T18:17:23.243 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:17:23.243 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.241+0000 7f33d70a5640 1 Processor -- start 2024-03-25T18:17:23.244 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.241+0000 7f33d70a5640 1 -- start start 2024-03-25T18:17:23.244 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.241+0000 7f33d70a5640 10 monclient: init 2024-03-25T18:17:23.244 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.242+0000 7f33d70a5640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:23.244 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.242+0000 7f33d70a5640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:17:23.244 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.242+0000 7f33d70a5640 10 monclient(hunting): picked mon.noname-c con 0x7f33d00a98f0 addr v1:172.21.15.98:6789/0 2024-03-25T18:17:23.244 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.242+0000 7f33d70a5640 10 monclient(hunting): picked mon.noname-b con 0x7f33d013f710 addr v1:172.21.15.97:6790/0 2024-03-25T18:17:23.244 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.242+0000 7f33d70a5640 10 monclient(hunting): picked mon.noname-a con 0x7f33d01430c0 addr v1:172.21.15.97:6789/0 2024-03-25T18:17:23.244 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.242+0000 7f33d70a5640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33d0142520 con 0x7f33d01430c0 2024-03-25T18:17:23.244 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.242+0000 7f33d70a5640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33d00b28a0 con 0x7f33d00a98f0 2024-03-25T18:17:23.244 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.242+0000 7f33d70a5640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33d00b2a30 con 0x7f33d013f710 2024-03-25T18:17:23.244 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.242+0000 7f33d70a5640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:23.244 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.242+0000 7f33d70a5640 10 monclient(hunting): authenticate will time out at 7655.518125s 2024-03-25T18:17:23.244 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.242+0000 7f33d4e41640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f33d00a98f0 0x7f33d00a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.97:60628/0 (socket says 172.21.15.97:60628) 2024-03-25T18:17:23.244 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.242+0000 7f33d4e41640 1 -- 172.21.15.97:0/3687259442 learned_addr learned my addr 172.21.15.97:0/3687259442 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:17:23.297 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.296+0000 7f33cf7fe640 1 -- 172.21.15.97:0/3687259442 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2244614925 0 0) 0x7f33d0142520 con 0x7f33d01430c0 2024-03-25T18:17:23.297 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.296+0000 7f33cf7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:17:23.297 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.296+0000 7f33cf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:23.297 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.296+0000 7f33cf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:23.297 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.296+0000 7f33cf7fe640 10 monclient(hunting): my global_id is 4266 2024-03-25T18:17:23.297 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.296+0000 7f33cf7fe640 1 -- 172.21.15.97:0/3687259442 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33bc003520 con 0x7f33d01430c0 2024-03-25T18:17:23.317 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.316+0000 7f33cf7fe640 1 -- 172.21.15.97:0/3687259442 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1088212935 0 0) 0x7f33bc003520 con 0x7f33d01430c0 2024-03-25T18:17:23.317 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.316+0000 7f33cf7fe640 1 -- 172.21.15.97:0/3687259442 >> v1:172.21.15.97:6790/0 conn(0x7f33d013f710 legacy=0x7f33d01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:23.318 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.316+0000 7f33cf7fe640 1 -- 172.21.15.97:0/3687259442 >> v1:172.21.15.98:6789/0 conn(0x7f33d00a98f0 legacy=0x7f33d00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:23.318 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.316+0000 7f33cf7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:17:23.318 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.316+0000 7f33cf7fe640 1 monclient: found mon.noname-a 2024-03-25T18:17:23.318 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.316+0000 7f33cf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:23.318 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.316+0000 7f33cf7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.97:6789/0 2024-03-25T18:17:23.318 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.316+0000 7f33cf7fe640 1 -- 172.21.15.97:0/3687259442 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33d00b44c0 con 0x7f33d01430c0 2024-03-25T18:17:23.318 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.316+0000 7f33cf7fe640 10 monclient: _finish_auth 0 2024-03-25T18:17:23.318 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.316+0000 7f33cf7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:17:23.318 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.316+0000 7f33cf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:23.318 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.316+0000 7f33cf7fe640 1 -- 172.21.15.97:0/3687259442 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f33b8001690 con 0x7f33d01430c0 2024-03-25T18:17:23.318 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.316+0000 7f33cf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:23.318 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.316+0000 7f33cf7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:17:23.318 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.316+0000 7f33cf7fe640 10 monclient: dump: 2024-03-25T18:17:23.319 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:17:23.319 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:17:23.319 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:17:23.319 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:17:23.319 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:23.319 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:17:23.319 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:17:23.319 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:17:23.319 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:17:23.319 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:17:23.319 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.316+0000 7f33d70a5640 5 monclient: authenticate success, global_id 4266 2024-03-25T18:17:23.319 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.316+0000 7f33d70a5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:17:23.324 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.323+0000 7f33cf7fe640 1 -- 172.21.15.97:0/3687259442 <== mon.0 v1:172.21.15.97:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f33b80017f0 con 0x7f33d01430c0 2024-03-25T18:17:23.324 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.323+0000 7f33cf7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:23.325 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.323+0000 7f33cf7fe640 1 -- 172.21.15.97:0/3687259442 <== mon.0 v1:172.21.15.97:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f33b8001c40 con 0x7f33d01430c0 2024-03-25T18:17:23.325 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.323+0000 7f33cf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:23.325 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.323+0000 7f33cf7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:17:23.325 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.323+0000 7f33cf7fe640 10 monclient: dump: 2024-03-25T18:17:23.325 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:17:23.325 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:17:23.325 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:17:23.325 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:17:23.325 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:23.325 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:17:23.325 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:17:23.325 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:17:23.325 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:17:23.325 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:17:23.326 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.323+0000 7f33d70a5640 10 monclient: get_monmap_and_config success 2024-03-25T18:17:23.326 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.323+0000 7f33d70a5640 4 set_mon_vals no callback set 2024-03-25T18:17:23.327 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.324+0000 7f33d70a5640 10 monclient: shutdown 2024-03-25T18:17:23.328 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.324+0000 7f33d70a5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:23.328 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.324+0000 7f33d70a5640 1 -- 172.21.15.97:0/3687259442 >> v1:172.21.15.97:6789/0 conn(0x7f33d01430c0 legacy=0x7f33d014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:23.328 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.324+0000 7f33d70a5640 1 -- 172.21.15.97:0/3687259442 shutdown_connections 2024-03-25T18:17:23.328 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.324+0000 7f33d70a5640 1 -- 172.21.15.97:0/3687259442 >> 172.21.15.97:0/3687259442 conn(0x7f33d00b7ea0 msgr2=0x7f33d00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:23.328 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.324+0000 7f33d70a5640 1 -- 172.21.15.97:0/3687259442 shutdown_connections 2024-03-25T18:17:23.328 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.324+0000 7f33d70a5640 1 -- 172.21.15.97:0/3687259442 wait complete. 2024-03-25T18:17:23.328 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.325+0000 7f33d70a5640 10 monclient: build_initial_monmap 2024-03-25T18:17:23.328 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.325+0000 7f33d70a5640 10 monclient: monmap: 2024-03-25T18:17:23.328 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:17:23.328 INFO:teuthology.misc.health.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:17:23.328 INFO:teuthology.misc.health.smithi097.stderr:last_changed 0.000000 2024-03-25T18:17:23.328 INFO:teuthology.misc.health.smithi097.stderr:created 0.000000 2024-03-25T18:17:23.329 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:23.329 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 1 2024-03-25T18:17:23.329 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:17:23.329 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T18:17:23.329 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-c 2024-03-25T18:17:23.329 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:17:23.329 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.326+0000 7f33d70a5640 1 librados: starting msgr at 2024-03-25T18:17:23.329 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.326+0000 7f33d70a5640 1 librados: starting objecter 2024-03-25T18:17:23.329 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.326+0000 7f33d70a5640 1 Processor -- start 2024-03-25T18:17:23.329 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.326+0000 7f33d70a5640 1 -- start start 2024-03-25T18:17:23.329 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.326+0000 7f33d70a5640 1 librados: setting wanted keys 2024-03-25T18:17:23.329 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.326+0000 7f33d70a5640 1 librados: calling monclient init 2024-03-25T18:17:23.329 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.326+0000 7f33d70a5640 10 monclient: init 2024-03-25T18:17:23.330 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.327+0000 7f33d70a5640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:23.330 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.327+0000 7f33d70a5640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:17:23.331 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.327+0000 7f33d70a5640 10 monclient(hunting): picked mon.noname-a con 0x7f33d00a98f0 addr v1:172.21.15.97:6789/0 2024-03-25T18:17:23.331 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.327+0000 7f33d70a5640 10 monclient(hunting): picked mon.noname-b con 0x7f33d013f710 addr v1:172.21.15.98:6789/0 2024-03-25T18:17:23.331 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.327+0000 7f33d70a5640 10 monclient(hunting): picked mon.noname-c con 0x7f33d01430c0 addr v1:172.21.15.97:6790/0 2024-03-25T18:17:23.331 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.327+0000 7f33d70a5640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33d006fbb0 con 0x7f33d00a98f0 2024-03-25T18:17:23.331 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.327+0000 7f33d70a5640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33d006fe20 con 0x7f33d013f710 2024-03-25T18:17:23.331 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.327+0000 7f33d70a5640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33d0070100 con 0x7f33d01430c0 2024-03-25T18:17:23.331 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.327+0000 7f33d70a5640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:23.331 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.327+0000 7f33d70a5640 10 monclient(hunting): authenticate will time out at 7655.603528s 2024-03-25T18:17:23.340 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.339+0000 7f33d4e41640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f33d00a98f0 0x7f33d0070be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:56806/0 (socket says 172.21.15.97:56806) 2024-03-25T18:17:23.340 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.339+0000 7f33d4e41640 1 -- 172.21.15.97:0/381989655 learned_addr learned my addr 172.21.15.97:0/381989655 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:17:23.344 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.343+0000 7f33cd7fa640 10 client.?.objecter ms_handle_connect 0x7f33d00a98f0 2024-03-25T18:17:23.344 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.343+0000 7f33cd7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:23.355 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.354+0000 7f33cd7fa640 10 client.?.objecter ms_handle_connect 0x7f33d01430c0 2024-03-25T18:17:23.355 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.354+0000 7f33cd7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:23.359 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.358+0000 7f33cd7fa640 10 client.?.objecter ms_handle_connect 0x7f33d013f710 2024-03-25T18:17:23.359 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.358+0000 7f33cd7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:23.359 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.358+0000 7f33cd7fa640 1 -- 172.21.15.97:0/381989655 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1807359767 0 0) 0x7f33d006fbb0 con 0x7f33d00a98f0 2024-03-25T18:17:23.360 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.358+0000 7f33cd7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:17:23.360 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.358+0000 7f33cd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:23.360 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.358+0000 7f33cd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:23.360 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.358+0000 7f33cd7fa640 10 monclient(hunting): my global_id is 4269 2024-03-25T18:17:23.360 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.358+0000 7f33cd7fa640 1 -- 172.21.15.97:0/381989655 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33b0004eb0 con 0x7f33d00a98f0 2024-03-25T18:17:23.368 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.367+0000 7f33cd7fa640 1 -- 172.21.15.97:0/381989655 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3663622374 0 0) 0x7f33d0070100 con 0x7f33d01430c0 2024-03-25T18:17:23.368 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.367+0000 7f33cd7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:17:23.368 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.367+0000 7f33cd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:23.369 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.367+0000 7f33cd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:23.369 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.367+0000 7f33cd7fa640 10 monclient(hunting): my global_id is 4262 2024-03-25T18:17:23.369 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.367+0000 7f33cd7fa640 1 -- 172.21.15.97:0/381989655 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33d006fbb0 con 0x7f33d01430c0 2024-03-25T18:17:23.369 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.368+0000 7f33cd7fa640 1 -- 172.21.15.97:0/381989655 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1105011453 0 0) 0x7f33d006fe20 con 0x7f33d013f710 2024-03-25T18:17:23.369 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.368+0000 7f33cd7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:17:23.369 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.368+0000 7f33cd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:23.369 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.368+0000 7f33cd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:23.370 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.368+0000 7f33cd7fa640 10 monclient(hunting): my global_id is 4267 2024-03-25T18:17:23.370 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.368+0000 7f33cd7fa640 1 -- 172.21.15.97:0/381989655 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33d0070100 con 0x7f33d013f710 2024-03-25T18:17:23.374 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.372+0000 7f33cd7fa640 1 -- 172.21.15.97:0/381989655 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1182283415 0 0) 0x7f33b0004eb0 con 0x7f33d00a98f0 2024-03-25T18:17:23.374 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.372+0000 7f33cd7fa640 1 -- 172.21.15.97:0/381989655 --> v1:172.21.15.97:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f33d006fe20 con 0x7f33d00a98f0 2024-03-25T18:17:23.374 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.372+0000 7f33cd7fa640 1 -- 172.21.15.97:0/381989655 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f33c8001c30 con 0x7f33d00a98f0 2024-03-25T18:17:23.374 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.372+0000 7f33cd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:23.374 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.372+0000 7f33cd7fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:17:23.374 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.372+0000 7f33cd7fa640 10 monclient(hunting): dump: 2024-03-25T18:17:23.374 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:17:23.374 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:17:23.374 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:17:23.374 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:17:23.374 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:23.374 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:17:23.375 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:17:23.375 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:17:23.375 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:17:23.375 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:17:23.375 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.372+0000 7f33cd7fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:17:23.377 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.376+0000 7f33cd7fa640 1 -- 172.21.15.97:0/381989655 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1502255074 0 0) 0x7f33d0070100 con 0x7f33d013f710 2024-03-25T18:17:23.377 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.376+0000 7f33cd7fa640 1 -- 172.21.15.97:0/381989655 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f33b0004eb0 con 0x7f33d013f710 2024-03-25T18:17:23.378 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.376+0000 7f33cd7fa640 1 -- 172.21.15.97:0/381989655 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f33c0001860 con 0x7f33d013f710 2024-03-25T18:17:23.378 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.376+0000 7f33cd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:23.378 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.376+0000 7f33cd7fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:17:23.378 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.376+0000 7f33cd7fa640 10 monclient(hunting): dump: 2024-03-25T18:17:23.378 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:17:23.378 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:17:23.378 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:17:23.378 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:17:23.378 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:23.378 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:17:23.378 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:17:23.378 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:17:23.378 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:17:23.379 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:17:23.380 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.379+0000 7f33cd7fa640 1 -- 172.21.15.97:0/381989655 <== mon.2 v1:172.21.15.97:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 90126706 0 0) 0x7f33d006fbb0 con 0x7f33d01430c0 2024-03-25T18:17:23.380 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.379+0000 7f33cd7fa640 1 -- 172.21.15.97:0/381989655 --> v1:172.21.15.97:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f33d0070100 con 0x7f33d01430c0 2024-03-25T18:17:23.380 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.379+0000 7f33cd7fa640 1 -- 172.21.15.97:0/381989655 <== mon.2 v1:172.21.15.97:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f33b8001b40 con 0x7f33d01430c0 2024-03-25T18:17:23.380 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.379+0000 7f33cd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:23.380 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.379+0000 7f33cd7fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:17:23.380 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.379+0000 7f33cd7fa640 10 monclient(hunting): dump: 2024-03-25T18:17:23.380 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:17:23.380 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:17:23.381 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:17:23.381 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:17:23.381 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:23.381 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:17:23.381 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:17:23.381 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:17:23.381 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:17:23.381 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:17:23.390 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.388+0000 7f33cd7fa640 1 -- 172.21.15.97:0/381989655 <== mon.1 v1:172.21.15.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3521263767 0 0) 0x7f33b0004eb0 con 0x7f33d013f710 2024-03-25T18:17:23.390 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.388+0000 7f33cd7fa640 1 -- 172.21.15.97:0/381989655 >> v1:172.21.15.97:6790/0 conn(0x7f33d01430c0 legacy=0x7f33d01e6840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:23.390 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.388+0000 7f33cd7fa640 1 -- 172.21.15.97:0/381989655 >> v1:172.21.15.97:6789/0 conn(0x7f33d00a98f0 legacy=0x7f33d0070be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:23.390 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.388+0000 7f33cd7fa640 10 monclient: _finish_hunting 0 2024-03-25T18:17:23.390 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.388+0000 7f33cd7fa640 1 monclient: found mon.b 2024-03-25T18:17:23.390 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.388+0000 7f33cd7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:23.390 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.388+0000 7f33cd7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:17:23.390 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.388+0000 7f33cd7fa640 1 -- 172.21.15.97:0/381989655 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33d00703e0 con 0x7f33d013f710 2024-03-25T18:17:23.390 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.388+0000 7f33cd7fa640 10 monclient: _finish_auth 0 2024-03-25T18:17:23.390 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.388+0000 7f33cd7fa640 10 monclient: _check_auth_tickets 2024-03-25T18:17:23.390 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.388+0000 7f33cd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:23.390 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.388+0000 7f33d70a5640 5 monclient: authenticate success, global_id 4267 2024-03-25T18:17:23.390 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.388+0000 7f33d70a5640 10 monclient: _renew_subs 2024-03-25T18:17:23.390 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.388+0000 7f33d70a5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:17:23.391 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.388+0000 7f33d70a5640 1 -- 172.21.15.97:0/381989655 --> v1:172.21.15.98:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33d006f240 con 0x7f33d013f710 2024-03-25T18:17:23.391 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.388+0000 7f33d70a5640 10 client.4267.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:17:23.391 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.388+0000 7f33d70a5640 10 monclient: _renew_subs 2024-03-25T18:17:23.391 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.388+0000 7f33d70a5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:17:23.391 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.388+0000 7f33d70a5640 1 -- 172.21.15.97:0/381989655 --> v1:172.21.15.98:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f33d006e2f0 con 0x7f33d013f710 2024-03-25T18:17:23.391 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.388+0000 7f33d70a5640 1 librados: init done 2024-03-25T18:17:23.393 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.389+0000 7f33d70a5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:23.394 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.389+0000 7f33d70a5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:23.394 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.389+0000 7f33d70a5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:17:23.394 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.389+0000 7f33d70a5640 1 -- 172.21.15.97:0/381989655 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33d00627b0 con 0x7f33d013f710 2024-03-25T18:17:23.662 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.660+0000 7f33cd7fa640 1 -- 172.21.15.97:0/381989655 <== mon.1 v1:172.21.15.98:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f33c0001d30 con 0x7f33d013f710 2024-03-25T18:17:23.662 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.660+0000 7f33cd7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:23.662 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.660+0000 7f33cd7fa640 1 -- 172.21.15.97:0/381989655 <== mon.1 v1:172.21.15.98:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f33c0003790 con 0x7f33d013f710 2024-03-25T18:17:23.662 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.660+0000 7f33cd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:23.662 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.660+0000 7f33cd7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:17:23.662 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.660+0000 7f33cd7fa640 10 monclient: dump: 2024-03-25T18:17:23.662 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:17:23.662 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:17:23.662 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:17:23.662 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:17:23.662 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:23.662 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:17:23.663 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:17:23.663 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:17:23.663 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:17:23.663 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:17:23.663 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.660+0000 7f33ce7fc640 4 set_mon_vals no callback set 2024-03-25T18:17:23.663 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.661+0000 7f33cd7fa640 1 -- 172.21.15.97:0/381989655 <== mon.1 v1:172.21.15.98:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2377622137 0 0) 0x7f33c00197a0 con 0x7f33d013f710 2024-03-25T18:17:23.663 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.662+0000 7f33d4e41640 10 monclient: get_auth_request con 0x7f33b006e110 auth_method 0 2024-03-25T18:17:23.663 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.662+0000 7f33cd7fa640 10 client.4267.objecter ms_handle_connect 0x7f33b006e110 2024-03-25T18:17:23.668 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.667+0000 7f33cd7fa640 1 -- 172.21.15.97:0/381989655 <== mon.1 v1:172.21.15.98:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 430711918 0 0) 0x7f33c00534e0 con 0x7f33d013f710 2024-03-25T18:17:23.668 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.667+0000 7f33cd7fa640 10 client.4267.objecter ms_dispatch 0x7f33d0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:17:23.668 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.667+0000 7f33cd7fa640 3 client.4267.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:17:23.668 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.667+0000 7f33cd7fa640 3 client.4267.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:17:23.668 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.667+0000 7f33cd7fa640 20 client.4267.objecter dump_active .. 0 homeless 2024-03-25T18:17:23.670 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.669+0000 7f33cd7fa640 1 -- 172.21.15.97:0/381989655 <== mon.1 v1:172.21.15.98:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f33c004ee20 con 0x7f33d013f710 2024-03-25T18:17:23.670 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.669+0000 7f33cd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:23.670 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.669+0000 7f33cd7fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:17:23.926 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.924+0000 7f33d70a5640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:23.926 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.924+0000 7f33d70a5640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:23.926 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.924+0000 7f33d70a5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:17:23.926 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.924+0000 7f33d70a5640 1 -- 172.21.15.97:0/381989655 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f33d006f3d0 con 0x7f33d013f710 2024-03-25T18:17:23.947 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.946+0000 7f33cd7fa640 1 -- 172.21.15.97:0/381989655 <== mon.1 v1:172.21.15.98:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2169427412) 0x7f33c0050f20 con 0x7f33d013f710 2024-03-25T18:17:23.947 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.946+0000 7f33cd7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:23.948 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.946+0000 7f33cd7fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:17:23.948 INFO:teuthology.misc.health.smithi097.stdout: 2024-03-25T18:17:23.948 INFO:teuthology.misc.health.smithi097.stdout:{"epoch":10,"fsid":"4c3d472c-79b8-486b-9e3d-2738d4306754","created":"2024-03-25T18:16:14.737309+0000","modified":"2024-03-25T18:16:20.357492+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:20.357492+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cb5152b9-262f-45c3-94b8-18d0e81df558","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6daf8793-0838-473d-a7d6-9a0d2894c75d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"136f1493-3b9c-4f43-a2e7-641482d69dae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"397d538c-4f64-4daf-9e8b-662d09f91327","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4f1df08-663f-4bda-93f8-3878ab7c24b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c4b18d70-a3ce-4b15-9b70-eae5525bd761","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8dd4033a-de0e-4f8a-96dc-59f3c1f825b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0077722e-ffa6-4a3d-9891-929d9869b286","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:17:23.951 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.949+0000 7f33d70a5640 10 librados: watch_flush enter 2024-03-25T18:17:23.951 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.949+0000 7f33d70a5640 10 librados: watch_flush exit 2024-03-25T18:17:23.951 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.949+0000 7f33d70a5640 20 client.4267.objecter shutdown clearing up homeless session... 2024-03-25T18:17:23.951 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.949+0000 7f33d70a5640 10 client.4267.objecter successfully canceled tick 2024-03-25T18:17:23.951 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.949+0000 7f33d70a5640 1 -- 172.21.15.97:0/381989655 >> v1:172.21.15.98:6800/800387573 conn(0x7f33b006e110 legacy=0x7f33b00705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:23.951 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.949+0000 7f33d70a5640 10 monclient: shutdown 2024-03-25T18:17:23.951 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.949+0000 7f33d70a5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:23.951 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.949+0000 7f33d70a5640 1 -- 172.21.15.97:0/381989655 >> v1:172.21.15.98:6789/0 conn(0x7f33d013f710 legacy=0x7f33d00712f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:23.951 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.949+0000 7f33d70a5640 1 -- 172.21.15.97:0/381989655 shutdown_connections 2024-03-25T18:17:23.951 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.949+0000 7f33d70a5640 1 -- 172.21.15.97:0/381989655 >> 172.21.15.97:0/381989655 conn(0x7f33d00b7ea0 msgr2=0x7f33d0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:23.951 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.949+0000 7f33d70a5640 1 -- 172.21.15.97:0/381989655 shutdown_connections 2024-03-25T18:17:23.951 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.949+0000 7f33d70a5640 1 -- 172.21.15.97:0/381989655 wait complete. 2024-03-25T18:17:23.952 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:23.949+0000 7f33d70a5640 1 librados: shutdown 2024-03-25T18:17:23.963 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:17:30.769 DEBUG:teuthology.orchestra.run.smithi097:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:17:30.888 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:30.887+0000 7fddfc558640 10 monclient: get_monmap_and_config 2024-03-25T18:17:30.888 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:30.887+0000 7fddfc558640 10 monclient: build_initial_monmap 2024-03-25T18:17:30.888 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:30.887+0000 7fddfc558640 10 monclient: monmap: 2024-03-25T18:17:30.888 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:17:30.888 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:17:30.888 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:17:30.888484+0000 2024-03-25T18:17:30.888 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:17:30.888484+0000 2024-03-25T18:17:30.889 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:30.889 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:17:30.889 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:17:30.889 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T18:17:30.889 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-b 2024-03-25T18:17:30.889 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:17:30.890 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:30.887+0000 7fddfc558640 1 Processor -- start 2024-03-25T18:17:30.890 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:30.888+0000 7fddfc558640 1 -- start start 2024-03-25T18:17:30.890 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:30.888+0000 7fddfc558640 10 monclient: init 2024-03-25T18:17:30.890 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:30.889+0000 7fddfc558640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:30.890 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:30.889+0000 7fddfc558640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:17:30.890 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:30.889+0000 7fddfc558640 10 monclient(hunting): picked mon.noname-b con 0x7fddf40a98f0 addr v1:172.21.15.97:6790/0 2024-03-25T18:17:30.890 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:30.889+0000 7fddfc558640 10 monclient(hunting): picked mon.noname-c con 0x7fddf413f710 addr v1:172.21.15.98:6789/0 2024-03-25T18:17:30.890 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:30.889+0000 7fddfc558640 10 monclient(hunting): picked mon.noname-a con 0x7fddf41430c0 addr v1:172.21.15.97:6789/0 2024-03-25T18:17:30.890 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:30.889+0000 7fddfc558640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddf4142520 con 0x7fddf41430c0 2024-03-25T18:17:30.890 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:30.889+0000 7fddfc558640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddf40b28a0 con 0x7fddf413f710 2024-03-25T18:17:30.890 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:30.889+0000 7fddfc558640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddf40b2a30 con 0x7fddf40a98f0 2024-03-25T18:17:30.890 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:30.889+0000 7fddfc558640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:30.890 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:30.889+0000 7fddfc558640 10 monclient(hunting): authenticate will time out at 7663.165016s 2024-03-25T18:17:30.894 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:30.893+0000 7fddf9af3640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fddf413f710 0x7fddf41429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.97:46648/0 (socket says 172.21.15.97:46648) 2024-03-25T18:17:30.894 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:30.893+0000 7fddf9af3640 1 -- 172.21.15.97:0/1698540299 learned_addr learned my addr 172.21.15.97:0/1698540299 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:17:30.903 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:30.902+0000 7fddf92f2640 1 -- 172.21.15.97:0/1698540299 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2295193063 0 0) 0x7fddf40b28a0 con 0x7fddf413f710 2024-03-25T18:17:30.904 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:30.902+0000 7fddf92f2640 10 monclient(hunting): _negotiate 2024-03-25T18:17:30.904 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:30.902+0000 7fddf92f2640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:30.904 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:30.902+0000 7fddf92f2640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:30.904 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:30.902+0000 7fddf92f2640 10 monclient(hunting): my global_id is 4270 2024-03-25T18:17:30.904 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:30.902+0000 7fddf92f2640 1 -- 172.21.15.97:0/1698540299 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdde0003520 con 0x7fddf413f710 2024-03-25T18:17:30.914 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:30.913+0000 7fddf92f2640 1 -- 172.21.15.97:0/1698540299 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 664507460 0 0) 0x7fdde0003520 con 0x7fddf413f710 2024-03-25T18:17:30.914 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:30.913+0000 7fddf92f2640 1 -- 172.21.15.97:0/1698540299 >> v1:172.21.15.97:6790/0 conn(0x7fddf40a98f0 legacy=0x7fddf40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:30.914 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:30.913+0000 7fddf92f2640 1 -- 172.21.15.97:0/1698540299 >> v1:172.21.15.97:6789/0 conn(0x7fddf41430c0 legacy=0x7fddf414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:30.914 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:30.913+0000 7fddf92f2640 10 monclient: _finish_hunting 0 2024-03-25T18:17:30.914 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:30.913+0000 7fddf92f2640 1 monclient: found mon.noname-c 2024-03-25T18:17:30.915 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:30.913+0000 7fddf92f2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:30.915 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:30.913+0000 7fddf92f2640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.98:6789/0 2024-03-25T18:17:30.915 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:30.913+0000 7fddf92f2640 1 -- 172.21.15.97:0/1698540299 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddf40b44c0 con 0x7fddf413f710 2024-03-25T18:17:30.915 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:30.913+0000 7fddf92f2640 10 monclient: _finish_auth 0 2024-03-25T18:17:30.915 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:30.913+0000 7fddf92f2640 10 monclient: _check_auth_tickets 2024-03-25T18:17:30.915 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:30.913+0000 7fddf92f2640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:30.915 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:30.913+0000 7fddf92f2640 1 -- 172.21.15.97:0/1698540299 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7fdde4001690 con 0x7fddf413f710 2024-03-25T18:17:30.915 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:30.913+0000 7fddf92f2640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:30.915 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:30.913+0000 7fddf92f2640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:17:30.915 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:30.913+0000 7fddf92f2640 10 monclient: dump: 2024-03-25T18:17:30.915 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:17:30.915 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:17:30.915 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:17:30.915 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:17:30.915 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:30.915 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:17:30.916 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:17:30.916 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:17:30.916 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:17:30.916 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:17:30.916 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:30.913+0000 7fddfc558640 5 monclient: authenticate success, global_id 4270 2024-03-25T18:17:30.916 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:30.913+0000 7fddfc558640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:17:30.922 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:30.921+0000 7fddf92f2640 1 -- 172.21.15.97:0/1698540299 <== mon.1 v1:172.21.15.98:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdde40017f0 con 0x7fddf413f710 2024-03-25T18:17:30.922 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:30.921+0000 7fddf92f2640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:30.922 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:30.921+0000 7fddf92f2640 1 -- 172.21.15.97:0/1698540299 <== mon.1 v1:172.21.15.98:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7fdde4001c00 con 0x7fddf413f710 2024-03-25T18:17:30.923 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:30.921+0000 7fddfc558640 10 monclient: get_monmap_and_config success 2024-03-25T18:17:30.923 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:30.921+0000 7fddfc558640 4 set_mon_vals no callback set 2024-03-25T18:17:30.923 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:30.921+0000 7fddf92f2640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:30.923 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:30.921+0000 7fddf92f2640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:17:30.923 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:30.921+0000 7fddf92f2640 10 monclient: dump: 2024-03-25T18:17:30.923 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:17:30.923 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:17:30.923 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:17:30.923 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:17:30.923 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:30.923 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:17:30.923 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:17:30.923 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:17:30.923 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:17:30.923 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:17:30.925 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:30.922+0000 7fddfc558640 10 monclient: shutdown 2024-03-25T18:17:30.925 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:30.922+0000 7fddfc558640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:30.925 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:30.922+0000 7fddfc558640 1 -- 172.21.15.97:0/1698540299 >> v1:172.21.15.98:6789/0 conn(0x7fddf413f710 legacy=0x7fddf41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:30.925 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:30.922+0000 7fddfc558640 1 -- 172.21.15.97:0/1698540299 shutdown_connections 2024-03-25T18:17:30.925 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:30.922+0000 7fddfc558640 1 -- 172.21.15.97:0/1698540299 >> 172.21.15.97:0/1698540299 conn(0x7fddf40b7ea0 msgr2=0x7fddf40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:30.925 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:30.922+0000 7fddfc558640 1 -- 172.21.15.97:0/1698540299 shutdown_connections 2024-03-25T18:17:30.925 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:30.922+0000 7fddfc558640 1 -- 172.21.15.97:0/1698540299 wait complete. 2024-03-25T18:17:30.925 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:30.923+0000 7fddfc558640 10 monclient: build_initial_monmap 2024-03-25T18:17:30.925 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:30.923+0000 7fddfc558640 10 monclient: monmap: 2024-03-25T18:17:30.925 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:17:30.926 INFO:teuthology.misc.health.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:17:30.926 INFO:teuthology.misc.health.smithi097.stderr:last_changed 0.000000 2024-03-25T18:17:30.926 INFO:teuthology.misc.health.smithi097.stderr:created 0.000000 2024-03-25T18:17:30.926 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:30.926 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 1 2024-03-25T18:17:30.926 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:17:30.926 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T18:17:30.926 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-c 2024-03-25T18:17:30.926 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:17:30.926 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:30.923+0000 7fddfc558640 1 librados: starting msgr at 2024-03-25T18:17:30.926 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:30.923+0000 7fddfc558640 1 librados: starting objecter 2024-03-25T18:17:30.926 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:30.924+0000 7fddfc558640 1 Processor -- start 2024-03-25T18:17:30.926 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:30.924+0000 7fddfc558640 1 -- start start 2024-03-25T18:17:30.926 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:30.924+0000 7fddfc558640 1 librados: setting wanted keys 2024-03-25T18:17:30.927 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:30.924+0000 7fddfc558640 1 librados: calling monclient init 2024-03-25T18:17:30.927 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:30.924+0000 7fddfc558640 10 monclient: init 2024-03-25T18:17:30.928 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:30.924+0000 7fddfc558640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:30.928 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:30.924+0000 7fddfc558640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:17:30.928 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:30.924+0000 7fddfc558640 10 monclient(hunting): picked mon.noname-a con 0x7fddf40a98f0 addr v1:172.21.15.97:6789/0 2024-03-25T18:17:30.928 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:30.924+0000 7fddfc558640 10 monclient(hunting): picked mon.noname-b con 0x7fddf413f710 addr v1:172.21.15.98:6789/0 2024-03-25T18:17:30.928 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:30.925+0000 7fddfc558640 10 monclient(hunting): picked mon.noname-c con 0x7fddf41430c0 addr v1:172.21.15.97:6790/0 2024-03-25T18:17:30.928 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:30.925+0000 7fddfc558640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddf413be70 con 0x7fddf40a98f0 2024-03-25T18:17:30.928 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:30.925+0000 7fddfc558640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddf4131660 con 0x7fddf413f710 2024-03-25T18:17:30.928 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:30.925+0000 7fddfc558640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddf41317f0 con 0x7fddf41430c0 2024-03-25T18:17:30.928 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:30.925+0000 7fddfc558640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:30.928 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:30.925+0000 7fddfc558640 10 monclient(hunting): authenticate will time out at 7663.201031s 2024-03-25T18:17:30.931 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:30.930+0000 7fddfaaf5640 1 --1- >> v1:172.21.15.97:6790/0 conn(0x7fddf41430c0 0x7fddf41e2fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.97:6790/0 says I am v1:172.21.15.97:54548/0 (socket says 172.21.15.97:54548) 2024-03-25T18:17:30.931 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:30.930+0000 7fddfaaf5640 1 -- 172.21.15.97:0/662860706 learned_addr learned my addr 172.21.15.97:0/662860706 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:17:30.934 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:30.933+0000 7fddeaffd640 10 client.?.objecter ms_handle_connect 0x7fddf41430c0 2024-03-25T18:17:30.934 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:30.933+0000 7fddeaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:30.940 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:30.939+0000 7fddeaffd640 10 client.?.objecter ms_handle_connect 0x7fddf40a98f0 2024-03-25T18:17:30.940 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:30.939+0000 7fddeaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:30.941 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:30.940+0000 7fddeaffd640 10 client.?.objecter ms_handle_connect 0x7fddf413f710 2024-03-25T18:17:30.942 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:30.940+0000 7fddeaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:30.944 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:30.943+0000 7fddeaffd640 1 -- 172.21.15.97:0/662860706 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4137814588 0 0) 0x7fddf41317f0 con 0x7fddf41430c0 2024-03-25T18:17:30.944 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:30.943+0000 7fddeaffd640 10 monclient(hunting): _negotiate 2024-03-25T18:17:30.944 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:30.943+0000 7fddeaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:30.944 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:30.943+0000 7fddeaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:30.944 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:30.943+0000 7fddeaffd640 10 monclient(hunting): my global_id is 4268 2024-03-25T18:17:30.945 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:30.943+0000 7fddeaffd640 1 -- 172.21.15.97:0/662860706 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fddd8004eb0 con 0x7fddf41430c0 2024-03-25T18:17:30.950 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:30.949+0000 7fddeaffd640 1 -- 172.21.15.97:0/662860706 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2528270397 0 0) 0x7fddf413be70 con 0x7fddf40a98f0 2024-03-25T18:17:30.951 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:30.949+0000 7fddeaffd640 10 monclient(hunting): _negotiate 2024-03-25T18:17:30.951 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:30.949+0000 7fddeaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:30.951 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:30.949+0000 7fddeaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:30.951 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:30.949+0000 7fddeaffd640 10 monclient(hunting): my global_id is 4275 2024-03-25T18:17:30.951 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:30.949+0000 7fddeaffd640 1 -- 172.21.15.97:0/662860706 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fddf41317f0 con 0x7fddf40a98f0 2024-03-25T18:17:30.956 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:30.954+0000 7fddeaffd640 1 -- 172.21.15.97:0/662860706 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2824558247 0 0) 0x7fddf4131660 con 0x7fddf413f710 2024-03-25T18:17:30.956 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:30.954+0000 7fddeaffd640 10 monclient(hunting): _negotiate 2024-03-25T18:17:30.956 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:30.954+0000 7fddeaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:30.956 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:30.954+0000 7fddeaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:30.956 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:30.954+0000 7fddeaffd640 10 monclient(hunting): my global_id is 4273 2024-03-25T18:17:30.956 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:30.954+0000 7fddeaffd640 1 -- 172.21.15.97:0/662860706 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fddf413be70 con 0x7fddf413f710 2024-03-25T18:17:30.956 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:30.955+0000 7fddeaffd640 1 -- 172.21.15.97:0/662860706 <== mon.2 v1:172.21.15.97:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2093839288 0 0) 0x7fddd8004eb0 con 0x7fddf41430c0 2024-03-25T18:17:30.956 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:30.955+0000 7fddeaffd640 1 -- 172.21.15.97:0/662860706 --> v1:172.21.15.97:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fddf4131660 con 0x7fddf41430c0 2024-03-25T18:17:30.956 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:30.955+0000 7fddeaffd640 1 -- 172.21.15.97:0/662860706 <== mon.2 v1:172.21.15.97:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7fddec001c90 con 0x7fddf41430c0 2024-03-25T18:17:30.956 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:30.955+0000 7fddeaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:30.956 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:30.955+0000 7fddeaffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:17:30.957 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:30.955+0000 7fddeaffd640 10 monclient(hunting): dump: 2024-03-25T18:17:30.957 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:17:30.957 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:17:30.957 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:17:30.957 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:17:30.957 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:30.957 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:17:30.957 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:17:30.957 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:17:30.957 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:17:30.957 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:17:30.957 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:30.955+0000 7fddeaffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:17:30.960 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:30.958+0000 7fddeaffd640 1 -- 172.21.15.97:0/662860706 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1992448048 0 0) 0x7fddf41317f0 con 0x7fddf40a98f0 2024-03-25T18:17:30.960 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:30.958+0000 7fddeaffd640 1 -- 172.21.15.97:0/662860706 --> v1:172.21.15.97:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fddd8004eb0 con 0x7fddf40a98f0 2024-03-25T18:17:30.960 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:30.958+0000 7fddeaffd640 1 -- 172.21.15.97:0/662860706 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7fddf00018b0 con 0x7fddf40a98f0 2024-03-25T18:17:30.960 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:30.958+0000 7fddeaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:30.960 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:30.958+0000 7fddeaffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:17:30.960 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:30.958+0000 7fddeaffd640 10 monclient(hunting): dump: 2024-03-25T18:17:30.960 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:17:30.960 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:17:30.960 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:17:30.960 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:17:30.960 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:30.961 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:17:30.961 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:17:30.961 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:17:30.961 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:17:30.961 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:17:30.968 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:30.967+0000 7fddeaffd640 1 -- 172.21.15.97:0/662860706 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 366455688 0 0) 0x7fddf413be70 con 0x7fddf413f710 2024-03-25T18:17:30.969 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:30.967+0000 7fddeaffd640 1 -- 172.21.15.97:0/662860706 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fddf41317f0 con 0x7fddf413f710 2024-03-25T18:17:30.969 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:30.967+0000 7fddeaffd640 1 -- 172.21.15.97:0/662860706 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7fdde4001b00 con 0x7fddf413f710 2024-03-25T18:17:30.969 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:30.967+0000 7fddeaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:30.969 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:30.967+0000 7fddeaffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:17:30.969 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:30.967+0000 7fddeaffd640 10 monclient(hunting): dump: 2024-03-25T18:17:30.969 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:17:30.969 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:17:30.969 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:17:30.969 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:17:30.969 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:30.969 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:17:30.969 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:17:30.969 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:17:30.969 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:17:30.969 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:17:30.975 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:30.974+0000 7fddeaffd640 1 -- 172.21.15.97:0/662860706 <== mon.2 v1:172.21.15.97:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2792976112 0 0) 0x7fddf4131660 con 0x7fddf41430c0 2024-03-25T18:17:30.975 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:30.974+0000 7fddeaffd640 1 -- 172.21.15.97:0/662860706 >> v1:172.21.15.98:6789/0 conn(0x7fddf413f710 legacy=0x7fddf41d76e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:30.975 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:30.974+0000 7fddeaffd640 1 -- 172.21.15.97:0/662860706 >> v1:172.21.15.97:6789/0 conn(0x7fddf40a98f0 legacy=0x7fddf4130f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:30.975 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:30.974+0000 7fddeaffd640 10 monclient: _finish_hunting 0 2024-03-25T18:17:30.975 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:30.974+0000 7fddeaffd640 1 monclient: found mon.c 2024-03-25T18:17:30.976 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:30.974+0000 7fddeaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:30.976 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:30.974+0000 7fddeaffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.97:6790/0 2024-03-25T18:17:30.976 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:30.974+0000 7fddeaffd640 1 -- 172.21.15.97:0/662860706 --> v1:172.21.15.97:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddf4131980 con 0x7fddf41430c0 2024-03-25T18:17:30.976 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:30.974+0000 7fddeaffd640 10 monclient: _finish_auth 0 2024-03-25T18:17:30.976 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:30.974+0000 7fddeaffd640 10 monclient: _check_auth_tickets 2024-03-25T18:17:30.976 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:30.974+0000 7fddeaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:30.976 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:30.974+0000 7fddfc558640 5 monclient: authenticate success, global_id 4268 2024-03-25T18:17:30.976 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:30.974+0000 7fddfc558640 10 monclient: _renew_subs 2024-03-25T18:17:30.976 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:30.974+0000 7fddfc558640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.97:6790/0 2024-03-25T18:17:30.976 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:30.974+0000 7fddfc558640 1 -- 172.21.15.97:0/662860706 --> v1:172.21.15.97:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddf4135c60 con 0x7fddf41430c0 2024-03-25T18:17:30.976 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:30.974+0000 7fddfc558640 10 client.4268.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:17:30.976 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:30.974+0000 7fddfc558640 10 monclient: _renew_subs 2024-03-25T18:17:30.976 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:30.974+0000 7fddfc558640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.97:6790/0 2024-03-25T18:17:30.976 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:30.974+0000 7fddfc558640 1 -- 172.21.15.97:0/662860706 --> v1:172.21.15.97:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fddf40b7770 con 0x7fddf41430c0 2024-03-25T18:17:30.976 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:30.974+0000 7fddfc558640 1 librados: init done 2024-03-25T18:17:30.977 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:30.975+0000 7fddfc558640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:30.977 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:30.975+0000 7fddfc558640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:30.977 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:30.975+0000 7fddfc558640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.97:6790/0 2024-03-25T18:17:30.977 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:30.975+0000 7fddfc558640 1 -- 172.21.15.97:0/662860706 --> v1:172.21.15.97:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fddf4074cc0 con 0x7fddf41430c0 2024-03-25T18:17:31.278 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:31.277+0000 7fddeaffd640 1 -- 172.21.15.97:0/662860706 <== mon.2 v1:172.21.15.97:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fddec003990 con 0x7fddf41430c0 2024-03-25T18:17:31.278 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:31.277+0000 7fddeaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:31.278 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:31.277+0000 7fddeaffd640 1 -- 172.21.15.97:0/662860706 <== mon.2 v1:172.21.15.97:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7fddec003ce0 con 0x7fddf41430c0 2024-03-25T18:17:31.278 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:31.277+0000 7fddeaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:31.278 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:31.277+0000 7fddeaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:17:31.278 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:31.277+0000 7fddeaffd640 10 monclient: dump: 2024-03-25T18:17:31.278 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:17:31.278 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:17:31.278 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:17:31.279 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:17:31.279 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:31.279 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:17:31.279 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:17:31.279 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:17:31.279 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:17:31.279 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:17:31.279 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:31.277+0000 7fddf8af1640 4 set_mon_vals no callback set 2024-03-25T18:17:31.280 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:31.278+0000 7fddeaffd640 1 -- 172.21.15.97:0/662860706 <== mon.2 v1:172.21.15.97:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2377622137 0 0) 0x7fddec003f00 con 0x7fddf41430c0 2024-03-25T18:17:31.280 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:31.278+0000 7fddfa2f4640 10 monclient: get_auth_request con 0x7fddd806e1a0 auth_method 0 2024-03-25T18:17:31.280 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:31.278+0000 7fddeaffd640 10 client.4268.objecter ms_handle_connect 0x7fddd806e1a0 2024-03-25T18:17:31.284 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:31.283+0000 7fddeaffd640 1 -- 172.21.15.97:0/662860706 <== mon.2 v1:172.21.15.97:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 430711918 0 0) 0x7fddec054ae0 con 0x7fddf41430c0 2024-03-25T18:17:31.284 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:31.283+0000 7fddeaffd640 10 client.4268.objecter ms_dispatch 0x7fddf4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:17:31.285 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:31.283+0000 7fddeaffd640 3 client.4268.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:17:31.285 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:31.283+0000 7fddeaffd640 3 client.4268.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:17:31.285 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:31.283+0000 7fddeaffd640 20 client.4268.objecter dump_active .. 0 homeless 2024-03-25T18:17:31.285 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:31.283+0000 7fddeaffd640 1 -- 172.21.15.97:0/662860706 <== mon.2 v1:172.21.15.97:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fddec0505f0 con 0x7fddf41430c0 2024-03-25T18:17:31.285 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:31.283+0000 7fddeaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:31.285 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:31.283+0000 7fddeaffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:17:31.533 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:31.532+0000 7fddfc558640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:31.533 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:31.532+0000 7fddfc558640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:31.533 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:31.532+0000 7fddfc558640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.97:6790/0 2024-03-25T18:17:31.533 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:31.532+0000 7fddfc558640 1 -- 172.21.15.97:0/662860706 --> v1:172.21.15.97:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fddf41371d0 con 0x7fddf41430c0 2024-03-25T18:17:31.565 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:31.563+0000 7fddeaffd640 1 -- 172.21.15.97:0/662860706 <== mon.2 v1:172.21.15.97:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2169427412) 0x7fddec052520 con 0x7fddf41430c0 2024-03-25T18:17:31.565 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:31.563+0000 7fddeaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:31.565 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:31.563+0000 7fddeaffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:17:31.565 INFO:teuthology.misc.health.smithi097.stdout: 2024-03-25T18:17:31.565 INFO:teuthology.misc.health.smithi097.stdout:{"epoch":10,"fsid":"4c3d472c-79b8-486b-9e3d-2738d4306754","created":"2024-03-25T18:16:14.737309+0000","modified":"2024-03-25T18:16:20.357492+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:20.357492+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cb5152b9-262f-45c3-94b8-18d0e81df558","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6daf8793-0838-473d-a7d6-9a0d2894c75d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"136f1493-3b9c-4f43-a2e7-641482d69dae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"397d538c-4f64-4daf-9e8b-662d09f91327","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4f1df08-663f-4bda-93f8-3878ab7c24b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c4b18d70-a3ce-4b15-9b70-eae5525bd761","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8dd4033a-de0e-4f8a-96dc-59f3c1f825b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0077722e-ffa6-4a3d-9891-929d9869b286","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:17:31.568 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:31.566+0000 7fddfc558640 10 librados: watch_flush enter 2024-03-25T18:17:31.568 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:31.566+0000 7fddfc558640 10 librados: watch_flush exit 2024-03-25T18:17:31.568 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:31.566+0000 7fddfc558640 20 client.4268.objecter shutdown clearing up homeless session... 2024-03-25T18:17:31.568 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:31.566+0000 7fddfc558640 10 client.4268.objecter successfully canceled tick 2024-03-25T18:17:31.568 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:31.566+0000 7fddfc558640 1 -- 172.21.15.97:0/662860706 >> v1:172.21.15.98:6800/800387573 conn(0x7fddd806e1a0 legacy=0x7fddd8070660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:31.568 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:31.566+0000 7fddfc558640 10 monclient: shutdown 2024-03-25T18:17:31.568 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:31.566+0000 7fddfc558640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:31.568 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:31.566+0000 7fddfc558640 1 -- 172.21.15.97:0/662860706 >> v1:172.21.15.97:6790/0 conn(0x7fddf41430c0 legacy=0x7fddf41e2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:31.568 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:31.566+0000 7fddfc558640 1 -- 172.21.15.97:0/662860706 shutdown_connections 2024-03-25T18:17:31.568 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:31.566+0000 7fddfc558640 1 -- 172.21.15.97:0/662860706 >> 172.21.15.97:0/662860706 conn(0x7fddf40b7ea0 msgr2=0x7fddf4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:31.569 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:31.566+0000 7fddfc558640 1 -- 172.21.15.97:0/662860706 shutdown_connections 2024-03-25T18:17:31.569 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:31.566+0000 7fddfc558640 1 -- 172.21.15.97:0/662860706 wait complete. 2024-03-25T18:17:31.569 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:31.566+0000 7fddfc558640 1 librados: shutdown 2024-03-25T18:17:31.580 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:17:31.916 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:01:33.955 34931== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:31.965 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:01:34.004 34931== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:32.018 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:01:34.057 34931== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:32.029 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:01:34.068 34935== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:32.036 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:01:34.079 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:32.039 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:01:34.078 34935== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:32.042 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:01:34.085 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:32.095 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:01:34.138 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:32.096 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:01:34.135 34935== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:38.386 DEBUG:teuthology.orchestra.run.smithi097:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:17:38.488 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.487+0000 7f9af6495640 10 monclient: get_monmap_and_config 2024-03-25T18:17:38.488 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.487+0000 7f9af6495640 10 monclient: build_initial_monmap 2024-03-25T18:17:38.489 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.487+0000 7f9af6495640 10 monclient: monmap: 2024-03-25T18:17:38.489 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:17:38.489 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:17:38.489 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:17:38.488834+0000 2024-03-25T18:17:38.489 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:17:38.488834+0000 2024-03-25T18:17:38.489 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:38.489 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:17:38.489 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:17:38.489 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T18:17:38.489 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-b 2024-03-25T18:17:38.489 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:17:38.490 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.488+0000 7f9af6495640 1 Processor -- start 2024-03-25T18:17:38.490 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.488+0000 7f9af6495640 1 -- start start 2024-03-25T18:17:38.491 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.488+0000 7f9af6495640 10 monclient: init 2024-03-25T18:17:38.491 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.488+0000 7f9af6495640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:38.491 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.488+0000 7f9af6495640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:17:38.491 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.488+0000 7f9af6495640 10 monclient(hunting): picked mon.noname-a con 0x7f9af00a98f0 addr v1:172.21.15.97:6789/0 2024-03-25T18:17:38.491 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.488+0000 7f9af6495640 10 monclient(hunting): picked mon.noname-b con 0x7f9af013f710 addr v1:172.21.15.97:6790/0 2024-03-25T18:17:38.491 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.488+0000 7f9af6495640 10 monclient(hunting): picked mon.noname-c con 0x7f9af01430c0 addr v1:172.21.15.98:6789/0 2024-03-25T18:17:38.491 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.488+0000 7f9af6495640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9af0142520 con 0x7f9af00a98f0 2024-03-25T18:17:38.491 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.488+0000 7f9af6495640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9af00b28a0 con 0x7f9af01430c0 2024-03-25T18:17:38.491 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.488+0000 7f9af6495640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9af00b2a30 con 0x7f9af013f710 2024-03-25T18:17:38.491 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.488+0000 7f9af6495640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:38.491 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.488+0000 7f9af6495640 10 monclient(hunting): authenticate will time out at 7670.764812s 2024-03-25T18:17:38.503 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.503+0000 7f9af4a32640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f9af01430c0 0x7f9af014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.97:51002/0 (socket says 172.21.15.97:51002) 2024-03-25T18:17:38.503 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.503+0000 7f9af4a32640 1 -- 172.21.15.97:0/2498683902 learned_addr learned my addr 172.21.15.97:0/2498683902 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:17:38.531 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.530+0000 7f9aeeffd640 1 -- 172.21.15.97:0/2498683902 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2796533009 0 0) 0x7f9af0142520 con 0x7f9af00a98f0 2024-03-25T18:17:38.531 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.530+0000 7f9aeeffd640 10 monclient(hunting): _negotiate 2024-03-25T18:17:38.531 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.530+0000 7f9aeeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:38.532 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.530+0000 7f9aeeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:38.532 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.530+0000 7f9aeeffd640 10 monclient(hunting): my global_id is 4278 2024-03-25T18:17:38.532 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.530+0000 7f9aeeffd640 1 -- 172.21.15.97:0/2498683902 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ad8003520 con 0x7f9af00a98f0 2024-03-25T18:17:38.533 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.532+0000 7f9aeeffd640 1 -- 172.21.15.97:0/2498683902 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2565904689 0 0) 0x7f9af00b28a0 con 0x7f9af01430c0 2024-03-25T18:17:38.533 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.532+0000 7f9aeeffd640 10 monclient(hunting): _negotiate 2024-03-25T18:17:38.533 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.532+0000 7f9aeeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:38.534 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.532+0000 7f9aeeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:38.534 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.532+0000 7f9aeeffd640 10 monclient(hunting): my global_id is 4276 2024-03-25T18:17:38.534 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.532+0000 7f9aeeffd640 1 -- 172.21.15.97:0/2498683902 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9af0142520 con 0x7f9af01430c0 2024-03-25T18:17:38.541 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.539+0000 7f9aeeffd640 1 -- 172.21.15.97:0/2498683902 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1779972141 0 0) 0x7f9af00b2a30 con 0x7f9af013f710 2024-03-25T18:17:38.541 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.539+0000 7f9aeeffd640 10 monclient(hunting): _negotiate 2024-03-25T18:17:38.541 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.539+0000 7f9aeeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:38.541 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.539+0000 7f9aeeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:38.541 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.539+0000 7f9aeeffd640 10 monclient(hunting): my global_id is 4271 2024-03-25T18:17:38.541 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.539+0000 7f9aeeffd640 1 -- 172.21.15.97:0/2498683902 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9af00b28a0 con 0x7f9af013f710 2024-03-25T18:17:38.542 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.540+0000 7f9aeeffd640 1 -- 172.21.15.97:0/2498683902 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1660534467 0 0) 0x7f9ad8003520 con 0x7f9af00a98f0 2024-03-25T18:17:38.542 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.540+0000 7f9aeeffd640 1 -- 172.21.15.97:0/2498683902 >> v1:172.21.15.97:6790/0 conn(0x7f9af013f710 legacy=0x7f9af01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:38.542 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.540+0000 7f9aeeffd640 1 -- 172.21.15.97:0/2498683902 >> v1:172.21.15.98:6789/0 conn(0x7f9af01430c0 legacy=0x7f9af014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:38.542 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.540+0000 7f9aeeffd640 10 monclient: _finish_hunting 0 2024-03-25T18:17:38.542 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.540+0000 7f9aeeffd640 1 monclient: found mon.noname-a 2024-03-25T18:17:38.542 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.540+0000 7f9aeeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:38.542 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.540+0000 7f9aeeffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.97:6789/0 2024-03-25T18:17:38.542 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.540+0000 7f9aeeffd640 1 -- 172.21.15.97:0/2498683902 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9af00b44c0 con 0x7f9af00a98f0 2024-03-25T18:17:38.542 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.540+0000 7f9aeeffd640 10 monclient: _finish_auth 0 2024-03-25T18:17:38.542 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.540+0000 7f9aeeffd640 10 monclient: _check_auth_tickets 2024-03-25T18:17:38.543 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.540+0000 7f9aeeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:38.543 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.540+0000 7f9aeeffd640 1 -- 172.21.15.97:0/2498683902 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f9ae0001690 con 0x7f9af00a98f0 2024-03-25T18:17:38.543 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.540+0000 7f9aeeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:38.543 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.540+0000 7f9aeeffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:17:38.543 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.540+0000 7f9aeeffd640 10 monclient: dump: 2024-03-25T18:17:38.543 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:17:38.543 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:17:38.543 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:17:38.543 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:17:38.543 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:38.543 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:17:38.544 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:17:38.544 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:17:38.544 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:17:38.544 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:17:38.544 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.540+0000 7f9af6495640 5 monclient: authenticate success, global_id 4278 2024-03-25T18:17:38.544 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.540+0000 7f9af6495640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:17:38.552 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.551+0000 7f9aeeffd640 1 -- 172.21.15.97:0/2498683902 <== mon.0 v1:172.21.15.97:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9ae00017f0 con 0x7f9af00a98f0 2024-03-25T18:17:38.552 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.551+0000 7f9aeeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:38.553 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.551+0000 7f9aeeffd640 1 -- 172.21.15.97:0/2498683902 <== mon.0 v1:172.21.15.97:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f9ae0001c00 con 0x7f9af00a98f0 2024-03-25T18:17:38.553 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.551+0000 7f9af6495640 10 monclient: get_monmap_and_config success 2024-03-25T18:17:38.553 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.551+0000 7f9af6495640 4 set_mon_vals no callback set 2024-03-25T18:17:38.553 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.551+0000 7f9aeeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:38.553 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.551+0000 7f9aeeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:17:38.553 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.551+0000 7f9aeeffd640 10 monclient: dump: 2024-03-25T18:17:38.553 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:17:38.553 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:17:38.553 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:17:38.553 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:17:38.553 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:38.553 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:17:38.553 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:17:38.553 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:17:38.553 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:17:38.554 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:17:38.555 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.552+0000 7f9af6495640 10 monclient: shutdown 2024-03-25T18:17:38.555 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.552+0000 7f9af6495640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:38.555 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.552+0000 7f9af6495640 1 -- 172.21.15.97:0/2498683902 >> v1:172.21.15.97:6789/0 conn(0x7f9af00a98f0 legacy=0x7f9af00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:38.555 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.552+0000 7f9af6495640 1 -- 172.21.15.97:0/2498683902 shutdown_connections 2024-03-25T18:17:38.555 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.552+0000 7f9af6495640 1 -- 172.21.15.97:0/2498683902 >> 172.21.15.97:0/2498683902 conn(0x7f9af00b7ea0 msgr2=0x7f9af00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:38.555 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.553+0000 7f9af6495640 1 -- 172.21.15.97:0/2498683902 shutdown_connections 2024-03-25T18:17:38.555 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.553+0000 7f9af6495640 1 -- 172.21.15.97:0/2498683902 wait complete. 2024-03-25T18:17:38.555 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.553+0000 7f9af6495640 10 monclient: build_initial_monmap 2024-03-25T18:17:38.555 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.553+0000 7f9af6495640 10 monclient: monmap: 2024-03-25T18:17:38.556 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:17:38.556 INFO:teuthology.misc.health.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:17:38.556 INFO:teuthology.misc.health.smithi097.stderr:last_changed 0.000000 2024-03-25T18:17:38.556 INFO:teuthology.misc.health.smithi097.stderr:created 0.000000 2024-03-25T18:17:38.556 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:38.556 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 1 2024-03-25T18:17:38.556 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:17:38.556 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T18:17:38.556 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-c 2024-03-25T18:17:38.556 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:17:38.556 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.554+0000 7f9af6495640 1 librados: starting msgr at 2024-03-25T18:17:38.556 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.554+0000 7f9af6495640 1 librados: starting objecter 2024-03-25T18:17:38.556 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.554+0000 7f9af6495640 1 Processor -- start 2024-03-25T18:17:38.557 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.554+0000 7f9af6495640 1 -- start start 2024-03-25T18:17:38.557 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.554+0000 7f9af6495640 1 librados: setting wanted keys 2024-03-25T18:17:38.557 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.554+0000 7f9af6495640 1 librados: calling monclient init 2024-03-25T18:17:38.557 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.554+0000 7f9af6495640 10 monclient: init 2024-03-25T18:17:38.557 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.555+0000 7f9af6495640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:38.557 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.555+0000 7f9af6495640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:17:38.557 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.555+0000 7f9af6495640 10 monclient(hunting): picked mon.noname-b con 0x7f9af00a98f0 addr v1:172.21.15.98:6789/0 2024-03-25T18:17:38.558 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.555+0000 7f9af6495640 10 monclient(hunting): picked mon.noname-a con 0x7f9af013f710 addr v1:172.21.15.97:6789/0 2024-03-25T18:17:38.558 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.555+0000 7f9af6495640 10 monclient(hunting): picked mon.noname-c con 0x7f9af01430c0 addr v1:172.21.15.97:6790/0 2024-03-25T18:17:38.558 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.555+0000 7f9af6495640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9af01db050 con 0x7f9af013f710 2024-03-25T18:17:38.558 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.555+0000 7f9af6495640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9af01dab70 con 0x7f9af00a98f0 2024-03-25T18:17:38.558 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.555+0000 7f9af6495640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9af01dad00 con 0x7f9af01430c0 2024-03-25T18:17:38.558 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.555+0000 7f9af6495640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:38.558 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.555+0000 7f9af6495640 10 monclient(hunting): authenticate will time out at 7670.831386s 2024-03-25T18:17:38.578 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.577+0000 7f9aeffff640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f9af00a98f0 0x7f9af01db4e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.97:51056/0 (socket says 172.21.15.97:51056) 2024-03-25T18:17:38.579 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.577+0000 7f9aeffff640 1 -- 172.21.15.97:0/2841211459 learned_addr learned my addr 172.21.15.97:0/2841211459 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:17:38.579 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.578+0000 7f9aecff9640 10 client.?.objecter ms_handle_connect 0x7f9af00a98f0 2024-03-25T18:17:38.579 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.578+0000 7f9aecff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:38.580 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.579+0000 7f9aecff9640 10 client.?.objecter ms_handle_connect 0x7f9af01430c0 2024-03-25T18:17:38.580 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.579+0000 7f9aecff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:38.584 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.582+0000 7f9aecff9640 10 client.?.objecter ms_handle_connect 0x7f9af013f710 2024-03-25T18:17:38.584 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.582+0000 7f9aecff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:38.584 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.582+0000 7f9aecff9640 1 -- 172.21.15.97:0/2841211459 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2738724181 0 0) 0x7f9af01dab70 con 0x7f9af00a98f0 2024-03-25T18:17:38.584 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.582+0000 7f9aecff9640 10 monclient(hunting): _negotiate 2024-03-25T18:17:38.584 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.582+0000 7f9aecff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:38.584 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.582+0000 7f9aecff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:38.584 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.582+0000 7f9aecff9640 10 monclient(hunting): my global_id is 4279 2024-03-25T18:17:38.584 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.582+0000 7f9aecff9640 1 -- 172.21.15.97:0/2841211459 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9adc004eb0 con 0x7f9af00a98f0 2024-03-25T18:17:38.592 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.590+0000 7f9aecff9640 1 -- 172.21.15.97:0/2841211459 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1597979552 0 0) 0x7f9af01dad00 con 0x7f9af01430c0 2024-03-25T18:17:38.592 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.590+0000 7f9aecff9640 10 monclient(hunting): _negotiate 2024-03-25T18:17:38.592 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.590+0000 7f9aecff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:38.592 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.591+0000 7f9aecff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:38.592 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.591+0000 7f9aecff9640 10 monclient(hunting): my global_id is 4274 2024-03-25T18:17:38.592 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.591+0000 7f9aecff9640 1 -- 172.21.15.97:0/2841211459 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9af01dab70 con 0x7f9af01430c0 2024-03-25T18:17:38.602 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.601+0000 7f9aecff9640 1 -- 172.21.15.97:0/2841211459 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2981805222 0 0) 0x7f9af01db050 con 0x7f9af013f710 2024-03-25T18:17:38.602 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.601+0000 7f9aecff9640 10 monclient(hunting): _negotiate 2024-03-25T18:17:38.602 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.601+0000 7f9aecff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:38.602 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.601+0000 7f9aecff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:38.602 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.601+0000 7f9aecff9640 10 monclient(hunting): my global_id is 4281 2024-03-25T18:17:38.602 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.601+0000 7f9aecff9640 1 -- 172.21.15.97:0/2841211459 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9af01dad00 con 0x7f9af013f710 2024-03-25T18:17:38.604 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.603+0000 7f9aecff9640 1 -- 172.21.15.97:0/2841211459 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 477084957 0 0) 0x7f9adc004eb0 con 0x7f9af00a98f0 2024-03-25T18:17:38.604 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.603+0000 7f9aecff9640 1 -- 172.21.15.97:0/2841211459 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9af01db050 con 0x7f9af00a98f0 2024-03-25T18:17:38.604 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.603+0000 7f9aecff9640 1 -- 172.21.15.97:0/2841211459 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f9ae0001bc0 con 0x7f9af00a98f0 2024-03-25T18:17:38.604 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.603+0000 7f9aecff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:38.604 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.603+0000 7f9aecff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:17:38.604 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.603+0000 7f9aecff9640 10 monclient(hunting): dump: 2024-03-25T18:17:38.604 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:17:38.605 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:17:38.605 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:17:38.605 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:17:38.605 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:38.605 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:17:38.605 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:17:38.605 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:17:38.605 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:17:38.605 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:17:38.605 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.603+0000 7f9aecff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:17:38.607 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.605+0000 7f9aecff9640 1 -- 172.21.15.97:0/2841211459 <== mon.2 v1:172.21.15.97:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2474726373 0 0) 0x7f9af01dab70 con 0x7f9af01430c0 2024-03-25T18:17:38.607 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.605+0000 7f9aecff9640 1 -- 172.21.15.97:0/2841211459 --> v1:172.21.15.97:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9adc004eb0 con 0x7f9af01430c0 2024-03-25T18:17:38.607 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.605+0000 7f9aecff9640 1 -- 172.21.15.97:0/2841211459 <== mon.2 v1:172.21.15.97:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f9ae8001b50 con 0x7f9af01430c0 2024-03-25T18:17:38.607 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.605+0000 7f9aecff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:38.607 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.605+0000 7f9aecff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:17:38.607 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.605+0000 7f9aecff9640 10 monclient(hunting): dump: 2024-03-25T18:17:38.607 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:17:38.607 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:17:38.607 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:17:38.607 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:17:38.607 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:38.607 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:17:38.607 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:17:38.607 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:17:38.608 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:17:38.608 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:17:38.615 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.614+0000 7f9aecff9640 1 -- 172.21.15.97:0/2841211459 <== mon.2 v1:172.21.15.97:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2759080456 0 0) 0x7f9adc004eb0 con 0x7f9af01430c0 2024-03-25T18:17:38.616 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.614+0000 7f9aecff9640 1 -- 172.21.15.97:0/2841211459 >> v1:172.21.15.98:6789/0 conn(0x7f9af00a98f0 legacy=0x7f9af01db4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:38.616 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.614+0000 7f9aecff9640 1 -- 172.21.15.97:0/2841211459 >> v1:172.21.15.97:6789/0 conn(0x7f9af013f710 legacy=0x7f9af01d7670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:38.616 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.614+0000 7f9aecff9640 10 monclient: _finish_hunting 0 2024-03-25T18:17:38.616 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.614+0000 7f9aecff9640 1 monclient: found mon.c 2024-03-25T18:17:38.616 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.614+0000 7f9aecff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:38.616 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.614+0000 7f9aecff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.97:6790/0 2024-03-25T18:17:38.616 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.614+0000 7f9aecff9640 1 -- 172.21.15.97:0/2841211459 --> v1:172.21.15.97:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9af00b70a0 con 0x7f9af01430c0 2024-03-25T18:17:38.616 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.614+0000 7f9aecff9640 10 monclient: _finish_auth 0 2024-03-25T18:17:38.616 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.614+0000 7f9aecff9640 10 monclient: _check_auth_tickets 2024-03-25T18:17:38.616 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.614+0000 7f9aecff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:38.616 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.614+0000 7f9af6495640 5 monclient: authenticate success, global_id 4274 2024-03-25T18:17:38.616 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.614+0000 7f9af6495640 10 monclient: _renew_subs 2024-03-25T18:17:38.616 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.614+0000 7f9af6495640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.97:6790/0 2024-03-25T18:17:38.616 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.614+0000 7f9af6495640 1 -- 172.21.15.97:0/2841211459 --> v1:172.21.15.97:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9af00b72f0 con 0x7f9af01430c0 2024-03-25T18:17:38.617 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.614+0000 7f9af6495640 10 client.4274.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:17:38.617 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.614+0000 7f9af6495640 10 monclient: _renew_subs 2024-03-25T18:17:38.617 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.614+0000 7f9af6495640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.97:6790/0 2024-03-25T18:17:38.617 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.614+0000 7f9af6495640 1 -- 172.21.15.97:0/2841211459 --> v1:172.21.15.97:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9af01e32a0 con 0x7f9af01430c0 2024-03-25T18:17:38.617 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.614+0000 7f9af6495640 1 librados: init done 2024-03-25T18:17:38.617 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.615+0000 7f9af6495640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:38.617 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.615+0000 7f9af6495640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:38.617 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.615+0000 7f9af6495640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.97:6790/0 2024-03-25T18:17:38.617 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.615+0000 7f9af6495640 1 -- 172.21.15.97:0/2841211459 --> v1:172.21.15.97:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9af00627b0 con 0x7f9af01430c0 2024-03-25T18:17:38.921 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.920+0000 7f9aecff9640 1 -- 172.21.15.97:0/2841211459 <== mon.2 v1:172.21.15.97:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9ae8002060 con 0x7f9af01430c0 2024-03-25T18:17:38.922 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.920+0000 7f9aecff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:38.922 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.920+0000 7f9aecff9640 1 -- 172.21.15.97:0/2841211459 <== mon.2 v1:172.21.15.97:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f9ae8004d00 con 0x7f9af01430c0 2024-03-25T18:17:38.922 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.920+0000 7f9aecff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:38.922 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.920+0000 7f9aecff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:17:38.922 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.920+0000 7f9aecff9640 10 monclient: dump: 2024-03-25T18:17:38.922 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:17:38.922 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:17:38.922 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:17:38.922 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:17:38.922 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:38.923 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:17:38.923 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:17:38.923 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:17:38.923 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:17:38.923 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:17:38.923 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.920+0000 7f9aee7fc640 4 set_mon_vals no callback set 2024-03-25T18:17:38.923 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.921+0000 7f9aecff9640 1 -- 172.21.15.97:0/2841211459 <== mon.2 v1:172.21.15.97:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2377622137 0 0) 0x7f9ae801ad50 con 0x7f9af01430c0 2024-03-25T18:17:38.924 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.922+0000 7f9aeffff640 10 monclient: get_auth_request con 0x7f9adc06df20 auth_method 0 2024-03-25T18:17:38.924 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.922+0000 7f9aecff9640 10 client.4274.objecter ms_handle_connect 0x7f9adc06df20 2024-03-25T18:17:38.929 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.928+0000 7f9aecff9640 1 -- 172.21.15.97:0/2841211459 <== mon.2 v1:172.21.15.97:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 430711918 0 0) 0x7f9ae8054a50 con 0x7f9af01430c0 2024-03-25T18:17:38.929 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.928+0000 7f9aecff9640 10 client.4274.objecter ms_dispatch 0x7f9af0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:17:38.929 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.928+0000 7f9aecff9640 3 client.4274.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:17:38.929 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.928+0000 7f9aecff9640 3 client.4274.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:17:38.929 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.928+0000 7f9aecff9640 20 client.4274.objecter dump_active .. 0 homeless 2024-03-25T18:17:38.929 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.928+0000 7f9aecff9640 1 -- 172.21.15.97:0/2841211459 <== mon.2 v1:172.21.15.97:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9ae8050390 con 0x7f9af01430c0 2024-03-25T18:17:38.929 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.928+0000 7f9aecff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:38.929 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:38.928+0000 7f9aecff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:17:39.189 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:39.188+0000 7f9af6495640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:39.189 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:39.188+0000 7f9af6495640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:39.189 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:39.188+0000 7f9af6495640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.97:6790/0 2024-03-25T18:17:39.189 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:39.188+0000 7f9af6495640 1 -- 172.21.15.97:0/2841211459 --> v1:172.21.15.97:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9af0074cc0 con 0x7f9af01430c0 2024-03-25T18:17:39.220 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:39.219+0000 7f9aecff9640 1 -- 172.21.15.97:0/2841211459 <== mon.2 v1:172.21.15.97:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2169427412) 0x7f9ae8052490 con 0x7f9af01430c0 2024-03-25T18:17:39.221 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:39.219+0000 7f9aecff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:39.221 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:39.219+0000 7f9aecff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:17:39.221 INFO:teuthology.misc.health.smithi097.stdout: 2024-03-25T18:17:39.221 INFO:teuthology.misc.health.smithi097.stdout:{"epoch":10,"fsid":"4c3d472c-79b8-486b-9e3d-2738d4306754","created":"2024-03-25T18:16:14.737309+0000","modified":"2024-03-25T18:16:20.357492+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:20.357492+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cb5152b9-262f-45c3-94b8-18d0e81df558","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6daf8793-0838-473d-a7d6-9a0d2894c75d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"136f1493-3b9c-4f43-a2e7-641482d69dae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"397d538c-4f64-4daf-9e8b-662d09f91327","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4f1df08-663f-4bda-93f8-3878ab7c24b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c4b18d70-a3ce-4b15-9b70-eae5525bd761","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8dd4033a-de0e-4f8a-96dc-59f3c1f825b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0077722e-ffa6-4a3d-9891-929d9869b286","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:17:39.223 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:39.221+0000 7f9af6495640 10 librados: watch_flush enter 2024-03-25T18:17:39.224 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:39.222+0000 7f9af6495640 10 librados: watch_flush exit 2024-03-25T18:17:39.224 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:39.222+0000 7f9af6495640 20 client.4274.objecter shutdown clearing up homeless session... 2024-03-25T18:17:39.224 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:39.222+0000 7f9af6495640 10 client.4274.objecter successfully canceled tick 2024-03-25T18:17:39.224 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:39.222+0000 7f9af6495640 1 -- 172.21.15.97:0/2841211459 >> v1:172.21.15.98:6800/800387573 conn(0x7f9adc06df20 legacy=0x7f9adc0703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:39.224 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:39.222+0000 7f9af6495640 10 monclient: shutdown 2024-03-25T18:17:39.224 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:39.222+0000 7f9af6495640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:39.224 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:39.222+0000 7f9af6495640 1 -- 172.21.15.97:0/2841211459 >> v1:172.21.15.97:6790/0 conn(0x7f9af01430c0 legacy=0x7f9af01393b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:39.224 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:39.222+0000 7f9af6495640 1 -- 172.21.15.97:0/2841211459 shutdown_connections 2024-03-25T18:17:39.224 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:39.222+0000 7f9af6495640 1 -- 172.21.15.97:0/2841211459 >> 172.21.15.97:0/2841211459 conn(0x7f9af00b7ea0 msgr2=0x7f9af0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:39.224 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:39.222+0000 7f9af6495640 1 -- 172.21.15.97:0/2841211459 shutdown_connections 2024-03-25T18:17:39.224 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:39.222+0000 7f9af6495640 1 -- 172.21.15.97:0/2841211459 wait complete. 2024-03-25T18:17:39.224 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:39.222+0000 7f9af6495640 1 librados: shutdown 2024-03-25T18:17:39.236 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:17:46.041 DEBUG:teuthology.orchestra.run.smithi097:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:17:46.159 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.159+0000 7f66625cb640 10 monclient: get_monmap_and_config 2024-03-25T18:17:46.159 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.159+0000 7f66625cb640 10 monclient: build_initial_monmap 2024-03-25T18:17:46.159 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.159+0000 7f66625cb640 10 monclient: monmap: 2024-03-25T18:17:46.159 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:17:46.159 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:17:46.159 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:17:46.159595+0000 2024-03-25T18:17:46.159 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:17:46.159595+0000 2024-03-25T18:17:46.159 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:46.160 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:17:46.160 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:17:46.160 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T18:17:46.160 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-b 2024-03-25T18:17:46.160 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:17:46.160 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.160+0000 7f66625cb640 1 Processor -- start 2024-03-25T18:17:46.160 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.160+0000 7f66625cb640 1 -- start start 2024-03-25T18:17:46.160 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.160+0000 7f66625cb640 10 monclient: init 2024-03-25T18:17:46.161 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.161+0000 7f66625cb640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:46.162 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.161+0000 7f66625cb640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:17:46.162 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.161+0000 7f66625cb640 10 monclient(hunting): picked mon.noname-a con 0x7f665c0a98f0 addr v1:172.21.15.97:6789/0 2024-03-25T18:17:46.162 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.161+0000 7f66625cb640 10 monclient(hunting): picked mon.noname-b con 0x7f665c13f710 addr v1:172.21.15.97:6790/0 2024-03-25T18:17:46.162 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.161+0000 7f66625cb640 10 monclient(hunting): picked mon.noname-c con 0x7f665c1430c0 addr v1:172.21.15.98:6789/0 2024-03-25T18:17:46.162 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.161+0000 7f66625cb640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f665c142520 con 0x7f665c0a98f0 2024-03-25T18:17:46.162 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.161+0000 7f66625cb640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f665c0b28a0 con 0x7f665c1430c0 2024-03-25T18:17:46.162 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.161+0000 7f66625cb640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f665c0b2a30 con 0x7f665c13f710 2024-03-25T18:17:46.162 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.161+0000 7f66625cb640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:46.162 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.161+0000 7f66625cb640 10 monclient(hunting): authenticate will time out at 7678.436248s 2024-03-25T18:17:46.170 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.170+0000 7f665b7fe640 1 --1- >> v1:172.21.15.97:6790/0 conn(0x7f665c13f710 0x7f665c1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6790/0 says I am v1:172.21.15.97:37074/0 (socket says 172.21.15.97:37074) 2024-03-25T18:17:46.170 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.170+0000 7f665b7fe640 1 -- 172.21.15.97:0/3687290353 learned_addr learned my addr 172.21.15.97:0/3687290353 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:17:46.186 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.186+0000 7f665affd640 1 -- 172.21.15.97:0/3687290353 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 562855258 0 0) 0x7f665c142520 con 0x7f665c0a98f0 2024-03-25T18:17:46.186 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.186+0000 7f665affd640 10 monclient(hunting): _negotiate 2024-03-25T18:17:46.186 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.186+0000 7f665affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:46.187 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.186+0000 7f665affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:46.187 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.186+0000 7f665affd640 10 monclient(hunting): my global_id is 4284 2024-03-25T18:17:46.187 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.186+0000 7f665affd640 1 -- 172.21.15.97:0/3687290353 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6644003520 con 0x7f665c0a98f0 2024-03-25T18:17:46.190 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.190+0000 7f665affd640 1 -- 172.21.15.97:0/3687290353 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 230900893 0 0) 0x7f665c0b2a30 con 0x7f665c13f710 2024-03-25T18:17:46.190 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.190+0000 7f665affd640 10 monclient(hunting): _negotiate 2024-03-25T18:17:46.190 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.190+0000 7f665affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:46.190 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.190+0000 7f665affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:46.190 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.190+0000 7f665affd640 10 monclient(hunting): my global_id is 4277 2024-03-25T18:17:46.190 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.190+0000 7f665affd640 1 -- 172.21.15.97:0/3687290353 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f665c142520 con 0x7f665c13f710 2024-03-25T18:17:46.201 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.201+0000 7f665affd640 1 -- 172.21.15.97:0/3687290353 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1351807028 0 0) 0x7f665c0b28a0 con 0x7f665c1430c0 2024-03-25T18:17:46.202 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.201+0000 7f665affd640 10 monclient(hunting): _negotiate 2024-03-25T18:17:46.202 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.201+0000 7f665affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:46.202 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.201+0000 7f665affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:46.202 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.201+0000 7f665affd640 10 monclient(hunting): my global_id is 4282 2024-03-25T18:17:46.202 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.201+0000 7f665affd640 1 -- 172.21.15.97:0/3687290353 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f665c0b2a30 con 0x7f665c1430c0 2024-03-25T18:17:46.205 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.204+0000 7f665affd640 1 -- 172.21.15.97:0/3687290353 <== mon.2 v1:172.21.15.97:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1292747253 0 0) 0x7f665c142520 con 0x7f665c13f710 2024-03-25T18:17:46.205 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.204+0000 7f665affd640 1 -- 172.21.15.97:0/3687290353 >> v1:172.21.15.98:6789/0 conn(0x7f665c1430c0 legacy=0x7f665c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:46.205 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.204+0000 7f665affd640 1 -- 172.21.15.97:0/3687290353 >> v1:172.21.15.97:6789/0 conn(0x7f665c0a98f0 legacy=0x7f665c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:46.205 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.204+0000 7f665affd640 10 monclient: _finish_hunting 0 2024-03-25T18:17:46.205 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.204+0000 7f665affd640 1 monclient: found mon.noname-b 2024-03-25T18:17:46.205 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.204+0000 7f665affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:46.205 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.204+0000 7f665affd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.97:6790/0 2024-03-25T18:17:46.206 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.204+0000 7f665affd640 1 -- 172.21.15.97:0/3687290353 --> v1:172.21.15.97:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f665c0b44c0 con 0x7f665c13f710 2024-03-25T18:17:46.206 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.204+0000 7f665affd640 10 monclient: _finish_auth 0 2024-03-25T18:17:46.206 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.204+0000 7f665affd640 10 monclient: _check_auth_tickets 2024-03-25T18:17:46.206 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.204+0000 7f665affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:46.206 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.204+0000 7f665affd640 1 -- 172.21.15.97:0/3687290353 <== mon.2 v1:172.21.15.97:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f6650001670 con 0x7f665c13f710 2024-03-25T18:17:46.206 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.204+0000 7f665affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:46.206 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.204+0000 7f665affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:17:46.206 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.204+0000 7f665affd640 10 monclient: dump: 2024-03-25T18:17:46.206 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:17:46.206 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:17:46.206 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:17:46.206 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:17:46.206 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:46.206 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:17:46.207 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:17:46.207 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:17:46.207 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:17:46.207 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:17:46.207 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.204+0000 7f66625cb640 5 monclient: authenticate success, global_id 4277 2024-03-25T18:17:46.207 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.204+0000 7f66625cb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:17:46.216 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.216+0000 7f665affd640 1 -- 172.21.15.97:0/3687290353 <== mon.2 v1:172.21.15.97:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f66500017d0 con 0x7f665c13f710 2024-03-25T18:17:46.216 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.216+0000 7f665affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:46.217 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.216+0000 7f66625cb640 10 monclient: get_monmap_and_config success 2024-03-25T18:17:46.217 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.216+0000 7f66625cb640 4 set_mon_vals no callback set 2024-03-25T18:17:46.217 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.216+0000 7f665affd640 1 -- 172.21.15.97:0/3687290353 <== mon.2 v1:172.21.15.97:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f6650001be0 con 0x7f665c13f710 2024-03-25T18:17:46.217 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.216+0000 7f665affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:46.217 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.216+0000 7f665affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:17:46.217 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.216+0000 7f665affd640 10 monclient: dump: 2024-03-25T18:17:46.217 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:17:46.217 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:17:46.217 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:17:46.217 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:17:46.217 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:46.217 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:17:46.217 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:17:46.217 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:17:46.218 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:17:46.218 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:17:46.220 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.217+0000 7f66625cb640 10 monclient: shutdown 2024-03-25T18:17:46.220 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.217+0000 7f66625cb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:46.220 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.217+0000 7f66625cb640 1 -- 172.21.15.97:0/3687290353 >> v1:172.21.15.97:6790/0 conn(0x7f665c13f710 legacy=0x7f665c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:46.220 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.217+0000 7f66625cb640 1 -- 172.21.15.97:0/3687290353 shutdown_connections 2024-03-25T18:17:46.220 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.217+0000 7f66625cb640 1 -- 172.21.15.97:0/3687290353 >> 172.21.15.97:0/3687290353 conn(0x7f665c0b7ea0 msgr2=0x7f665c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:46.220 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.217+0000 7f66625cb640 1 -- 172.21.15.97:0/3687290353 shutdown_connections 2024-03-25T18:17:46.220 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.217+0000 7f66625cb640 1 -- 172.21.15.97:0/3687290353 wait complete. 2024-03-25T18:17:46.220 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.218+0000 7f66625cb640 10 monclient: build_initial_monmap 2024-03-25T18:17:46.220 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.218+0000 7f66625cb640 10 monclient: monmap: 2024-03-25T18:17:46.220 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:17:46.221 INFO:teuthology.misc.health.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:17:46.221 INFO:teuthology.misc.health.smithi097.stderr:last_changed 0.000000 2024-03-25T18:17:46.221 INFO:teuthology.misc.health.smithi097.stderr:created 0.000000 2024-03-25T18:17:46.221 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:46.221 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 1 2024-03-25T18:17:46.221 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:17:46.221 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T18:17:46.221 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-c 2024-03-25T18:17:46.221 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:17:46.221 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.218+0000 7f66625cb640 1 librados: starting msgr at 2024-03-25T18:17:46.221 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.218+0000 7f66625cb640 1 librados: starting objecter 2024-03-25T18:17:46.221 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.218+0000 7f66625cb640 1 Processor -- start 2024-03-25T18:17:46.221 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.219+0000 7f66625cb640 1 -- start start 2024-03-25T18:17:46.221 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.219+0000 7f66625cb640 1 librados: setting wanted keys 2024-03-25T18:17:46.221 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.219+0000 7f66625cb640 1 librados: calling monclient init 2024-03-25T18:17:46.221 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.219+0000 7f66625cb640 10 monclient: init 2024-03-25T18:17:46.222 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.219+0000 7f66625cb640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:46.222 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.219+0000 7f66625cb640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:17:46.222 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.219+0000 7f66625cb640 10 monclient(hunting): picked mon.noname-b con 0x7f665c0a98f0 addr v1:172.21.15.98:6789/0 2024-03-25T18:17:46.222 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.219+0000 7f66625cb640 10 monclient(hunting): picked mon.noname-a con 0x7f665c13f710 addr v1:172.21.15.97:6789/0 2024-03-25T18:17:46.222 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.219+0000 7f66625cb640 10 monclient(hunting): picked mon.noname-c con 0x7f665c1430c0 addr v1:172.21.15.97:6790/0 2024-03-25T18:17:46.222 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.219+0000 7f66625cb640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f665c138700 con 0x7f665c13f710 2024-03-25T18:17:46.222 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.219+0000 7f66625cb640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f665c138b90 con 0x7f665c0a98f0 2024-03-25T18:17:46.222 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.219+0000 7f66625cb640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f665c0b16d0 con 0x7f665c1430c0 2024-03-25T18:17:46.222 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.219+0000 7f66625cb640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:46.222 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.219+0000 7f66625cb640 10 monclient(hunting): authenticate will time out at 7678.494608s 2024-03-25T18:17:46.224 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.224+0000 7f6660b68640 1 --1- >> v1:172.21.15.97:6790/0 conn(0x7f665c1430c0 0x7f665c1e1dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.97:6790/0 says I am v1:172.21.15.97:37118/0 (socket says 172.21.15.97:37118) 2024-03-25T18:17:46.224 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.224+0000 7f6660b68640 1 -- 172.21.15.97:0/2491854616 learned_addr learned my addr 172.21.15.97:0/2491854616 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:17:46.229 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.229+0000 7f6658ff9640 10 client.?.objecter ms_handle_connect 0x7f665c1430c0 2024-03-25T18:17:46.230 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.229+0000 7f6658ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:46.233 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.233+0000 7f6658ff9640 10 client.?.objecter ms_handle_connect 0x7f665c0a98f0 2024-03-25T18:17:46.233 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.233+0000 7f6658ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:46.242 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.242+0000 7f6658ff9640 10 client.?.objecter ms_handle_connect 0x7f665c13f710 2024-03-25T18:17:46.242 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.242+0000 7f6658ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:46.243 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.243+0000 7f6658ff9640 1 -- 172.21.15.97:0/2491854616 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4084066734 0 0) 0x7f665c138b90 con 0x7f665c0a98f0 2024-03-25T18:17:46.243 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.243+0000 7f6658ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:17:46.243 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.243+0000 7f6658ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:46.243 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.243+0000 7f6658ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:46.243 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.243+0000 7f6658ff9640 10 monclient(hunting): my global_id is 4285 2024-03-25T18:17:46.243 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.243+0000 7f6658ff9640 1 -- 172.21.15.97:0/2491854616 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6648004eb0 con 0x7f665c0a98f0 2024-03-25T18:17:46.259 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.259+0000 7f6658ff9640 1 -- 172.21.15.97:0/2491854616 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3571451345 0 0) 0x7f665c138700 con 0x7f665c13f710 2024-03-25T18:17:46.259 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.259+0000 7f6658ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:17:46.259 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.259+0000 7f6658ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:46.259 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.259+0000 7f6658ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:46.259 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.259+0000 7f6658ff9640 10 monclient(hunting): my global_id is 4287 2024-03-25T18:17:46.259 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.259+0000 7f6658ff9640 1 -- 172.21.15.97:0/2491854616 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f665c138b90 con 0x7f665c13f710 2024-03-25T18:17:46.336 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.336+0000 7f6658ff9640 1 -- 172.21.15.97:0/2491854616 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3290438062 0 0) 0x7f665c138b90 con 0x7f665c13f710 2024-03-25T18:17:46.336 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.336+0000 7f6658ff9640 1 -- 172.21.15.97:0/2491854616 --> v1:172.21.15.97:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f665c138700 con 0x7f665c13f710 2024-03-25T18:17:46.337 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.336+0000 7f6658ff9640 1 -- 172.21.15.97:0/2491854616 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f6650001ae0 con 0x7f665c13f710 2024-03-25T18:17:46.337 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.336+0000 7f6658ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:46.337 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.336+0000 7f6658ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:17:46.337 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.336+0000 7f6658ff9640 10 monclient(hunting): dump: 2024-03-25T18:17:46.337 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:17:46.337 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:17:46.337 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:17:46.337 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:17:46.337 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:46.337 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:17:46.337 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:17:46.337 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:17:46.337 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:17:46.337 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:17:46.338 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.336+0000 7f6658ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:17:46.353 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.352+0000 7f6658ff9640 1 -- 172.21.15.97:0/2491854616 <== mon.0 v1:172.21.15.97:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 5802260 0 0) 0x7f665c138700 con 0x7f665c13f710 2024-03-25T18:17:46.353 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.352+0000 7f6658ff9640 1 -- 172.21.15.97:0/2491854616 >> v1:172.21.15.97:6790/0 conn(0x7f665c1430c0 legacy=0x7f665c1e1dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:46.353 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.352+0000 7f6658ff9640 1 -- 172.21.15.97:0/2491854616 >> v1:172.21.15.98:6789/0 conn(0x7f665c0a98f0 legacy=0x7f665c137820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:46.353 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.352+0000 7f6658ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:17:46.353 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.352+0000 7f6658ff9640 1 monclient: found mon.a 2024-03-25T18:17:46.353 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.352+0000 7f6658ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:46.353 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.352+0000 7f6658ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:17:46.353 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.352+0000 7f6658ff9640 1 -- 172.21.15.97:0/2491854616 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f665c138050 con 0x7f665c13f710 2024-03-25T18:17:46.353 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.352+0000 7f6658ff9640 10 monclient: _finish_auth 0 2024-03-25T18:17:46.353 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.352+0000 7f6658ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:17:46.353 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.352+0000 7f6658ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:46.353 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.352+0000 7f66625cb640 5 monclient: authenticate success, global_id 4287 2024-03-25T18:17:46.353 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.352+0000 7f66625cb640 10 monclient: _renew_subs 2024-03-25T18:17:46.353 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.352+0000 7f66625cb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:17:46.354 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.352+0000 7f66625cb640 1 -- 172.21.15.97:0/2491854616 --> v1:172.21.15.97:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f665c1382a0 con 0x7f665c13f710 2024-03-25T18:17:46.354 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.352+0000 7f66625cb640 10 client.4287.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:17:46.354 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.352+0000 7f66625cb640 10 monclient: _renew_subs 2024-03-25T18:17:46.354 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.352+0000 7f66625cb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:17:46.354 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.352+0000 7f66625cb640 1 -- 172.21.15.97:0/2491854616 --> v1:172.21.15.97:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f665c1db170 con 0x7f665c13f710 2024-03-25T18:17:46.354 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.352+0000 7f66625cb640 1 librados: init done 2024-03-25T18:17:46.354 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.353+0000 7f66625cb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:46.355 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.353+0000 7f66625cb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:46.355 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.353+0000 7f66625cb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:17:46.355 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.353+0000 7f66625cb640 1 -- 172.21.15.97:0/2491854616 --> v1:172.21.15.97:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f665c0b3ef0 con 0x7f665c13f710 2024-03-25T18:17:46.634 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.634+0000 7f6658ff9640 1 -- 172.21.15.97:0/2491854616 <== mon.0 v1:172.21.15.97:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6650001210 con 0x7f665c13f710 2024-03-25T18:17:46.635 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.634+0000 7f6658ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:46.635 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.634+0000 7f6658ff9640 1 -- 172.21.15.97:0/2491854616 <== mon.0 v1:172.21.15.97:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f6650004c30 con 0x7f665c13f710 2024-03-25T18:17:46.635 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.634+0000 7f6658ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:46.635 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.634+0000 7f6658ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:17:46.635 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.634+0000 7f6658ff9640 10 monclient: dump: 2024-03-25T18:17:46.635 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:17:46.635 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:17:46.635 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:17:46.635 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:17:46.635 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:46.635 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:17:46.636 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:17:46.636 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:17:46.636 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:17:46.636 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:17:46.636 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.634+0000 7f665a7fc640 4 set_mon_vals no callback set 2024-03-25T18:17:46.636 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.635+0000 7f6658ff9640 1 -- 172.21.15.97:0/2491854616 <== mon.0 v1:172.21.15.97:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2377622137 0 0) 0x7f665001ac40 con 0x7f665c13f710 2024-03-25T18:17:46.636 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.636+0000 7f665bfff640 10 monclient: get_auth_request con 0x7f664806dd90 auth_method 0 2024-03-25T18:17:46.636 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.636+0000 7f6658ff9640 10 client.4287.objecter ms_handle_connect 0x7f664806dd90 2024-03-25T18:17:46.642 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.641+0000 7f6658ff9640 1 -- 172.21.15.97:0/2491854616 <== mon.0 v1:172.21.15.97:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 430711918 0 0) 0x7f6650054850 con 0x7f665c13f710 2024-03-25T18:17:46.642 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.641+0000 7f6658ff9640 10 client.4287.objecter ms_dispatch 0x7f665c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:17:46.642 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.641+0000 7f6658ff9640 3 client.4287.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:17:46.642 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.641+0000 7f6658ff9640 3 client.4287.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:17:46.642 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.641+0000 7f6658ff9640 20 client.4287.objecter dump_active .. 0 homeless 2024-03-25T18:17:46.642 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.642+0000 7f6658ff9640 1 -- 172.21.15.97:0/2491854616 <== mon.0 v1:172.21.15.97:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6650050190 con 0x7f665c13f710 2024-03-25T18:17:46.642 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.642+0000 7f6658ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:46.642 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.642+0000 7f6658ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:17:46.900 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.899+0000 7f66625cb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:46.900 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.899+0000 7f66625cb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:46.900 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.899+0000 7f66625cb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:17:46.900 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.899+0000 7f66625cb640 1 -- 172.21.15.97:0/2491854616 --> v1:172.21.15.97:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f665c0627b0 con 0x7f665c13f710 2024-03-25T18:17:46.936 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.935+0000 7f6658ff9640 1 -- 172.21.15.97:0/2491854616 <== mon.0 v1:172.21.15.97:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2169427412) 0x7f6650052290 con 0x7f665c13f710 2024-03-25T18:17:46.936 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.936+0000 7f6658ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:46.936 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.936+0000 7f6658ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:17:46.936 INFO:teuthology.misc.health.smithi097.stdout: 2024-03-25T18:17:46.937 INFO:teuthology.misc.health.smithi097.stdout:{"epoch":10,"fsid":"4c3d472c-79b8-486b-9e3d-2738d4306754","created":"2024-03-25T18:16:14.737309+0000","modified":"2024-03-25T18:16:20.357492+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:20.357492+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cb5152b9-262f-45c3-94b8-18d0e81df558","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6daf8793-0838-473d-a7d6-9a0d2894c75d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"136f1493-3b9c-4f43-a2e7-641482d69dae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"397d538c-4f64-4daf-9e8b-662d09f91327","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4f1df08-663f-4bda-93f8-3878ab7c24b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c4b18d70-a3ce-4b15-9b70-eae5525bd761","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8dd4033a-de0e-4f8a-96dc-59f3c1f825b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0077722e-ffa6-4a3d-9891-929d9869b286","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:17:46.939 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.938+0000 7f66625cb640 10 librados: watch_flush enter 2024-03-25T18:17:46.939 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.938+0000 7f66625cb640 10 librados: watch_flush exit 2024-03-25T18:17:46.939 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.938+0000 7f66625cb640 20 client.4287.objecter shutdown clearing up homeless session... 2024-03-25T18:17:46.939 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.938+0000 7f66625cb640 10 client.4287.objecter successfully canceled tick 2024-03-25T18:17:46.939 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.938+0000 7f66625cb640 1 -- 172.21.15.97:0/2491854616 >> v1:172.21.15.98:6800/800387573 conn(0x7f664806dd90 legacy=0x7f6648070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:46.939 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.938+0000 7f66625cb640 10 monclient: shutdown 2024-03-25T18:17:46.939 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.938+0000 7f66625cb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:46.939 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.938+0000 7f66625cb640 1 -- 172.21.15.97:0/2491854616 >> v1:172.21.15.97:6789/0 conn(0x7f665c13f710 legacy=0x7f665c1da880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:46.939 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.938+0000 7f66625cb640 1 -- 172.21.15.97:0/2491854616 shutdown_connections 2024-03-25T18:17:46.939 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.938+0000 7f66625cb640 1 -- 172.21.15.97:0/2491854616 >> 172.21.15.97:0/2491854616 conn(0x7f665c0b7ea0 msgr2=0x7f665c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:46.939 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.938+0000 7f66625cb640 1 -- 172.21.15.97:0/2491854616 shutdown_connections 2024-03-25T18:17:46.939 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.938+0000 7f66625cb640 1 -- 172.21.15.97:0/2491854616 wait complete. 2024-03-25T18:17:46.940 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:46.938+0000 7f66625cb640 1 librados: shutdown 2024-03-25T18:17:46.950 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:17:51.951 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:01:53.989 34931== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:51.997 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:01:54.040 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:51.998 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:01:54.041 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:52.008 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:01:54.047 34931== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:52.031 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:01:54.075 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:52.034 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:01:54.073 34931== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:52.075 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:01:54.114 34935== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:52.082 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:01:54.121 34935== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:52.110 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:01:54.149 34935== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:53.755 DEBUG:teuthology.orchestra.run.smithi097:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:17:53.855 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:53.854+0000 7fddfcf02640 10 monclient: get_monmap_and_config 2024-03-25T18:17:53.856 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:53.854+0000 7fddfcf02640 10 monclient: build_initial_monmap 2024-03-25T18:17:53.856 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:53.854+0000 7fddfcf02640 10 monclient: monmap: 2024-03-25T18:17:53.856 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:17:53.856 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:17:53.856 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:17:53.855850+0000 2024-03-25T18:17:53.856 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:17:53.855850+0000 2024-03-25T18:17:53.856 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:53.856 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:17:53.856 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:17:53.856 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T18:17:53.856 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-b 2024-03-25T18:17:53.856 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:17:53.857 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:53.855+0000 7fddfcf02640 1 Processor -- start 2024-03-25T18:17:53.858 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:53.855+0000 7fddfcf02640 1 -- start start 2024-03-25T18:17:53.858 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:53.855+0000 7fddfcf02640 10 monclient: init 2024-03-25T18:17:53.858 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:53.856+0000 7fddfcf02640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:53.858 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:53.856+0000 7fddfcf02640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:17:53.858 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:53.856+0000 7fddfcf02640 10 monclient(hunting): picked mon.noname-a con 0x7fddf80b4d60 addr v1:172.21.15.97:6789/0 2024-03-25T18:17:53.858 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:53.856+0000 7fddfcf02640 10 monclient(hunting): picked mon.noname-b con 0x7fddf80b1730 addr v1:172.21.15.97:6790/0 2024-03-25T18:17:53.858 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:53.856+0000 7fddfcf02640 10 monclient(hunting): picked mon.noname-c con 0x7fddf814be90 addr v1:172.21.15.98:6789/0 2024-03-25T18:17:53.858 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:53.856+0000 7fddfcf02640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddf80af940 con 0x7fddf80b4d60 2024-03-25T18:17:53.858 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:53.856+0000 7fddfcf02640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddf80b0a70 con 0x7fddf814be90 2024-03-25T18:17:53.858 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:53.856+0000 7fddfcf02640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddf80b0d50 con 0x7fddf80b1730 2024-03-25T18:17:53.858 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:53.856+0000 7fddfcf02640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:53.858 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:53.856+0000 7fddfcf02640 10 monclient(hunting): authenticate will time out at 7686.132039s 2024-03-25T18:17:53.867 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:53.865+0000 7fddf659c640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fddf80b4d60 0x7fddf813f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:41836/0 (socket says 172.21.15.97:41836) 2024-03-25T18:17:53.867 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:53.865+0000 7fddf659c640 1 -- 172.21.15.97:0/4224172090 learned_addr learned my addr 172.21.15.97:0/4224172090 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:17:53.883 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:53.882+0000 7fddf5d9b640 1 -- 172.21.15.97:0/4224172090 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2726774201 0 0) 0x7fddf80af940 con 0x7fddf80b4d60 2024-03-25T18:17:53.883 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:53.882+0000 7fddf5d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:17:53.883 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:53.882+0000 7fddf5d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:53.883 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:53.882+0000 7fddf5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:53.883 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:53.882+0000 7fddf5d9b640 10 monclient(hunting): my global_id is 4290 2024-03-25T18:17:53.883 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:53.882+0000 7fddf5d9b640 1 -- 172.21.15.97:0/4224172090 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdde4003520 con 0x7fddf80b4d60 2024-03-25T18:17:53.905 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:53.904+0000 7fddf5d9b640 1 -- 172.21.15.97:0/4224172090 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1012021725 0 0) 0x7fdde4003520 con 0x7fddf80b4d60 2024-03-25T18:17:53.905 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:53.904+0000 7fddf5d9b640 1 -- 172.21.15.97:0/4224172090 >> v1:172.21.15.97:6790/0 conn(0x7fddf80b1730 legacy=0x7fddf80b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:17:53.905 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:53.904+0000 7fddf5d9b640 1 -- 172.21.15.97:0/4224172090 >> v1:172.21.15.98:6789/0 conn(0x7fddf814be90 legacy=0x7fddf814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:17:53.905 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:53.904+0000 7fddf5d9b640 10 monclient: _finish_hunting 0 2024-03-25T18:17:53.905 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:53.904+0000 7fddf5d9b640 1 monclient: found mon.noname-a 2024-03-25T18:17:53.906 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:53.904+0000 7fddf5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:53.906 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:53.904+0000 7fddf5d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.97:6789/0 2024-03-25T18:17:53.906 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:53.904+0000 7fddf5d9b640 1 -- 172.21.15.97:0/4224172090 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddf80b1030 con 0x7fddf80b4d60 2024-03-25T18:17:53.906 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:53.904+0000 7fddf5d9b640 10 monclient: _finish_auth 0 2024-03-25T18:17:53.906 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:53.904+0000 7fddf5d9b640 10 monclient: _check_auth_tickets 2024-03-25T18:17:53.906 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:53.904+0000 7fddf5d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:53.906 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:53.904+0000 7fddf5d9b640 1 -- 172.21.15.97:0/4224172090 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7fdde8001690 con 0x7fddf80b4d60 2024-03-25T18:17:53.906 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:53.904+0000 7fddfcf02640 5 monclient: authenticate success, global_id 4290 2024-03-25T18:17:53.906 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:53.904+0000 7fddfcf02640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:17:53.906 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:53.904+0000 7fddf5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:53.906 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:53.904+0000 7fddf5d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:17:53.906 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:53.904+0000 7fddf5d9b640 10 monclient: dump: 2024-03-25T18:17:53.906 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:17:53.906 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:17:53.906 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:17:53.907 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:17:53.907 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:53.907 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:17:53.907 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:17:53.907 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:17:53.907 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:17:53.907 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:17:53.907 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:53.904+0000 7fddfcf02640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:17:53.935 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:53.933+0000 7fddf5d9b640 1 -- 172.21.15.97:0/4224172090 <== mon.0 v1:172.21.15.97:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdde8001860 con 0x7fddf80b4d60 2024-03-25T18:17:53.935 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:53.933+0000 7fddf5d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:53.935 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:53.933+0000 7fddf5d9b640 1 -- 172.21.15.97:0/4224172090 <== mon.0 v1:172.21.15.97:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7fdde8001cb0 con 0x7fddf80b4d60 2024-03-25T18:17:53.935 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:53.933+0000 7fddfcf02640 10 monclient: get_monmap_and_config success 2024-03-25T18:17:53.935 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:53.933+0000 7fddfcf02640 4 set_mon_vals no callback set 2024-03-25T18:17:53.935 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:53.933+0000 7fddf5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:53.935 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:53.933+0000 7fddf5d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:17:53.935 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:53.933+0000 7fddf5d9b640 10 monclient: dump: 2024-03-25T18:17:53.935 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:17:53.935 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:17:53.935 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:17:53.935 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:17:53.936 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:53.936 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:17:53.936 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:17:53.936 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:17:53.936 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:17:53.936 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:17:53.938 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:53.934+0000 7fddfcf02640 10 monclient: shutdown 2024-03-25T18:17:53.938 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:53.934+0000 7fddfcf02640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:53.938 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:53.934+0000 7fddfcf02640 1 -- 172.21.15.97:0/4224172090 >> v1:172.21.15.97:6789/0 conn(0x7fddf80b4d60 legacy=0x7fddf813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:53.938 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:53.934+0000 7fddfcf02640 1 -- 172.21.15.97:0/4224172090 shutdown_connections 2024-03-25T18:17:53.938 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:53.934+0000 7fddfcf02640 1 -- 172.21.15.97:0/4224172090 >> 172.21.15.97:0/4224172090 conn(0x7fddf80617a0 msgr2=0x7fddf8061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:53.938 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:53.934+0000 7fddfcf02640 1 -- 172.21.15.97:0/4224172090 shutdown_connections 2024-03-25T18:17:53.938 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:53.934+0000 7fddfcf02640 1 -- 172.21.15.97:0/4224172090 wait complete. 2024-03-25T18:17:53.939 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:53.935+0000 7fddfcf02640 10 monclient: build_initial_monmap 2024-03-25T18:17:53.939 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:53.935+0000 7fddfcf02640 10 monclient: monmap: 2024-03-25T18:17:53.939 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:17:53.939 INFO:teuthology.misc.health.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:17:53.939 INFO:teuthology.misc.health.smithi097.stderr:last_changed 0.000000 2024-03-25T18:17:53.939 INFO:teuthology.misc.health.smithi097.stderr:created 0.000000 2024-03-25T18:17:53.939 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:53.939 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 1 2024-03-25T18:17:53.939 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:17:53.939 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T18:17:53.939 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-c 2024-03-25T18:17:53.939 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:17:53.939 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:53.935+0000 7fddfcf02640 1 librados: starting msgr at 2024-03-25T18:17:53.939 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:53.935+0000 7fddfcf02640 1 librados: starting objecter 2024-03-25T18:17:53.939 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:53.935+0000 7fddfcf02640 1 Processor -- start 2024-03-25T18:17:53.940 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:53.935+0000 7fddfcf02640 1 -- start start 2024-03-25T18:17:53.940 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:53.935+0000 7fddfcf02640 1 librados: setting wanted keys 2024-03-25T18:17:53.940 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:53.935+0000 7fddfcf02640 1 librados: calling monclient init 2024-03-25T18:17:53.940 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:53.935+0000 7fddfcf02640 10 monclient: init 2024-03-25T18:17:53.940 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:53.936+0000 7fddfcf02640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:53.940 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:53.936+0000 7fddfcf02640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:17:53.940 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:53.936+0000 7fddfcf02640 10 monclient(hunting): picked mon.noname-a con 0x7fddf80b1730 addr v1:172.21.15.97:6789/0 2024-03-25T18:17:53.940 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:53.936+0000 7fddfcf02640 10 monclient(hunting): picked mon.noname-b con 0x7fddf80b4d60 addr v1:172.21.15.98:6789/0 2024-03-25T18:17:53.940 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:53.936+0000 7fddfcf02640 10 monclient(hunting): picked mon.noname-c con 0x7fddf814be90 addr v1:172.21.15.97:6790/0 2024-03-25T18:17:53.940 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:53.936+0000 7fddfcf02640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddf81385c0 con 0x7fddf80b1730 2024-03-25T18:17:53.940 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:53.936+0000 7fddfcf02640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddf8135f60 con 0x7fddf80b4d60 2024-03-25T18:17:53.940 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:53.936+0000 7fddfcf02640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddf81360f0 con 0x7fddf814be90 2024-03-25T18:17:53.940 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:53.936+0000 7fddfcf02640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:53.940 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:53.936+0000 7fddfcf02640 10 monclient(hunting): authenticate will time out at 7686.212310s 2024-03-25T18:17:53.940 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:53.937+0000 7fddf6d9d640 1 --1- >> v1:172.21.15.97:6790/0 conn(0x7fddf814be90 0x7fddf81e2e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.97:6790/0 says I am v1:172.21.15.97:49990/0 (socket says 172.21.15.97:49990) 2024-03-25T18:17:53.940 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:53.937+0000 7fddf6d9d640 1 -- 172.21.15.97:0/3179888891 learned_addr learned my addr 172.21.15.97:0/3179888891 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:17:53.941 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:53.937+0000 7fddeeffd640 10 client.?.objecter ms_handle_connect 0x7fddf814be90 2024-03-25T18:17:53.941 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:53.937+0000 7fddeeffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:53.941 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:53.938+0000 7fddeeffd640 10 client.?.objecter ms_handle_connect 0x7fddf80b4d60 2024-03-25T18:17:53.941 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:53.938+0000 7fddeeffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:53.959 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:53.958+0000 7fddeeffd640 10 client.?.objecter ms_handle_connect 0x7fddf80b1730 2024-03-25T18:17:53.995 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:53.958+0000 7fddeeffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:54.012 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:54.011+0000 7fddeeffd640 1 -- 172.21.15.97:0/3179888891 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1733091563 0 0) 0x7fddf81385c0 con 0x7fddf80b1730 2024-03-25T18:17:54.012 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:54.011+0000 7fddeeffd640 10 monclient(hunting): _negotiate 2024-03-25T18:17:54.013 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:54.011+0000 7fddeeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:54.013 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:54.011+0000 7fddeeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:54.013 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:54.011+0000 7fddeeffd640 10 monclient(hunting): my global_id is 4293 2024-03-25T18:17:54.013 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:54.011+0000 7fddeeffd640 1 -- 172.21.15.97:0/3179888891 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fddd8004eb0 con 0x7fddf80b1730 2024-03-25T18:17:54.020 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:54.019+0000 7fddeeffd640 1 -- 172.21.15.97:0/3179888891 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 706201117 0 0) 0x7fddf8135f60 con 0x7fddf80b4d60 2024-03-25T18:17:54.020 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:54.019+0000 7fddeeffd640 10 monclient(hunting): _negotiate 2024-03-25T18:17:54.020 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:54.019+0000 7fddeeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:54.020 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:54.019+0000 7fddeeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:54.021 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:54.019+0000 7fddeeffd640 10 monclient(hunting): my global_id is 4288 2024-03-25T18:17:54.021 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:54.019+0000 7fddeeffd640 1 -- 172.21.15.97:0/3179888891 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fddf81385c0 con 0x7fddf80b4d60 2024-03-25T18:17:54.026 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:54.025+0000 7fddeeffd640 1 -- 172.21.15.97:0/3179888891 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3970652346 0 0) 0x7fddd8004eb0 con 0x7fddf80b1730 2024-03-25T18:17:54.026 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:54.025+0000 7fddeeffd640 1 -- 172.21.15.97:0/3179888891 --> v1:172.21.15.97:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fddf8135f60 con 0x7fddf80b1730 2024-03-25T18:17:54.027 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:54.025+0000 7fddeeffd640 1 -- 172.21.15.97:0/3179888891 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7fdde8001b70 con 0x7fddf80b1730 2024-03-25T18:17:54.027 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:54.025+0000 7fddeeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:54.027 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:54.025+0000 7fddeeffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:17:54.027 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:54.025+0000 7fddeeffd640 10 monclient(hunting): dump: 2024-03-25T18:17:54.027 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:17:54.027 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:17:54.027 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:17:54.027 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:17:54.027 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:54.027 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:17:54.027 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:17:54.027 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:17:54.027 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:17:54.027 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:17:54.028 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:54.025+0000 7fddeeffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:17:54.041 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:54.039+0000 7fddeeffd640 1 -- 172.21.15.97:0/3179888891 <== mon.0 v1:172.21.15.97:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 182101221 0 0) 0x7fddf8135f60 con 0x7fddf80b1730 2024-03-25T18:17:54.041 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:54.039+0000 7fddeeffd640 1 -- 172.21.15.97:0/3179888891 >> v1:172.21.15.97:6790/0 conn(0x7fddf814be90 legacy=0x7fddf81e2e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:54.041 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:54.039+0000 7fddeeffd640 1 -- 172.21.15.97:0/3179888891 >> v1:172.21.15.98:6789/0 conn(0x7fddf80b4d60 legacy=0x7fddf812fef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:54.041 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:54.039+0000 7fddeeffd640 10 monclient: _finish_hunting 0 2024-03-25T18:17:54.041 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:54.039+0000 7fddeeffd640 1 monclient: found mon.a 2024-03-25T18:17:54.041 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:54.039+0000 7fddeeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:54.041 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:54.039+0000 7fddeeffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:17:54.041 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:54.039+0000 7fddeeffd640 1 -- 172.21.15.97:0/3179888891 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddf8143900 con 0x7fddf80b1730 2024-03-25T18:17:54.041 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:54.039+0000 7fddeeffd640 10 monclient: _finish_auth 0 2024-03-25T18:17:54.042 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:54.039+0000 7fddeeffd640 10 monclient: _check_auth_tickets 2024-03-25T18:17:54.042 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:54.039+0000 7fddeeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:54.042 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:54.039+0000 7fddfcf02640 5 monclient: authenticate success, global_id 4293 2024-03-25T18:17:54.042 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:54.039+0000 7fddfcf02640 10 monclient: _renew_subs 2024-03-25T18:17:54.042 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:54.039+0000 7fddfcf02640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:17:54.042 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:54.039+0000 7fddfcf02640 1 -- 172.21.15.97:0/3179888891 --> v1:172.21.15.97:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddf81d82b0 con 0x7fddf80b1730 2024-03-25T18:17:54.042 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:54.040+0000 7fddfcf02640 10 client.4293.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:17:54.042 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:54.040+0000 7fddfcf02640 10 monclient: _renew_subs 2024-03-25T18:17:54.042 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:54.040+0000 7fddfcf02640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:17:54.042 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:54.040+0000 7fddfcf02640 1 -- 172.21.15.97:0/3179888891 --> v1:172.21.15.97:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fddf81307e0 con 0x7fddf80b1730 2024-03-25T18:17:54.042 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:54.040+0000 7fddfcf02640 1 librados: init done 2024-03-25T18:17:54.043 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:54.040+0000 7fddfcf02640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:54.043 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:54.040+0000 7fddfcf02640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:54.043 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:54.040+0000 7fddfcf02640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:17:54.043 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:54.040+0000 7fddfcf02640 1 -- 172.21.15.97:0/3179888891 --> v1:172.21.15.97:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fddf80b22f0 con 0x7fddf80b1730 2024-03-25T18:17:54.357 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:54.356+0000 7fddeeffd640 1 -- 172.21.15.97:0/3179888891 <== mon.0 v1:172.21.15.97:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdde8001d70 con 0x7fddf80b1730 2024-03-25T18:17:54.357 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:54.356+0000 7fddeeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:54.358 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:54.356+0000 7fddeeffd640 1 -- 172.21.15.97:0/3179888891 <== mon.0 v1:172.21.15.97:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7fdde8004bf0 con 0x7fddf80b1730 2024-03-25T18:17:54.358 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:54.356+0000 7fddeeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:54.358 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:54.356+0000 7fddeeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:17:54.358 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:54.356+0000 7fddeeffd640 10 monclient: dump: 2024-03-25T18:17:54.358 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:17:54.358 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:17:54.358 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:17:54.358 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:17:54.358 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:54.358 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:17:54.358 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:17:54.358 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:17:54.358 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:17:54.359 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:17:54.359 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:54.356+0000 7fddf559a640 4 set_mon_vals no callback set 2024-03-25T18:17:54.359 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:54.357+0000 7fddeeffd640 1 -- 172.21.15.97:0/3179888891 <== mon.0 v1:172.21.15.97:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2377622137 0 0) 0x7fdde801abc0 con 0x7fddf80b1730 2024-03-25T18:17:54.359 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:54.357+0000 7fddeffff640 10 monclient: get_auth_request con 0x7fddd806dd50 auth_method 0 2024-03-25T18:17:54.359 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:54.358+0000 7fddeeffd640 10 client.4293.objecter ms_handle_connect 0x7fddd806dd50 2024-03-25T18:17:54.365 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:54.363+0000 7fddeeffd640 1 -- 172.21.15.97:0/3179888891 <== mon.0 v1:172.21.15.97:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 430711918 0 0) 0x7fdde8054750 con 0x7fddf80b1730 2024-03-25T18:17:54.365 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:54.363+0000 7fddeeffd640 10 client.4293.objecter ms_dispatch 0x7fddf8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:17:54.365 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:54.363+0000 7fddeeffd640 3 client.4293.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:17:54.365 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:54.363+0000 7fddeeffd640 3 client.4293.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:17:54.365 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:54.363+0000 7fddeeffd640 20 client.4293.objecter dump_active .. 0 homeless 2024-03-25T18:17:54.365 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:54.363+0000 7fddeeffd640 1 -- 172.21.15.97:0/3179888891 <== mon.0 v1:172.21.15.97:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdde8050090 con 0x7fddf80b1730 2024-03-25T18:17:54.365 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:54.363+0000 7fddeeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:54.365 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:54.363+0000 7fddeeffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:17:54.612 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:54.611+0000 7fddfcf02640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:54.612 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:54.611+0000 7fddfcf02640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:54.612 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:54.611+0000 7fddfcf02640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:17:54.613 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:54.611+0000 7fddfcf02640 1 -- 172.21.15.97:0/3179888891 --> v1:172.21.15.97:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fddf81383c0 con 0x7fddf80b1730 2024-03-25T18:17:54.657 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:54.656+0000 7fddeeffd640 1 -- 172.21.15.97:0/3179888891 <== mon.0 v1:172.21.15.97:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2169427412) 0x7fdde8052190 con 0x7fddf80b1730 2024-03-25T18:17:54.657 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:54.656+0000 7fddeeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:54.657 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:54.656+0000 7fddeeffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:17:54.658 INFO:teuthology.misc.health.smithi097.stdout: 2024-03-25T18:17:54.658 INFO:teuthology.misc.health.smithi097.stdout:{"epoch":10,"fsid":"4c3d472c-79b8-486b-9e3d-2738d4306754","created":"2024-03-25T18:16:14.737309+0000","modified":"2024-03-25T18:16:20.357492+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:20.357492+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cb5152b9-262f-45c3-94b8-18d0e81df558","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6daf8793-0838-473d-a7d6-9a0d2894c75d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"136f1493-3b9c-4f43-a2e7-641482d69dae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"397d538c-4f64-4daf-9e8b-662d09f91327","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4f1df08-663f-4bda-93f8-3878ab7c24b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c4b18d70-a3ce-4b15-9b70-eae5525bd761","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8dd4033a-de0e-4f8a-96dc-59f3c1f825b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0077722e-ffa6-4a3d-9891-929d9869b286","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:17:54.660 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:54.658+0000 7fddfcf02640 10 librados: watch_flush enter 2024-03-25T18:17:54.660 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:54.658+0000 7fddfcf02640 10 librados: watch_flush exit 2024-03-25T18:17:54.660 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:54.658+0000 7fddfcf02640 20 client.4293.objecter shutdown clearing up homeless session... 2024-03-25T18:17:54.660 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:54.658+0000 7fddfcf02640 10 client.4293.objecter successfully canceled tick 2024-03-25T18:17:54.660 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:54.658+0000 7fddfcf02640 1 -- 172.21.15.97:0/3179888891 >> v1:172.21.15.98:6800/800387573 conn(0x7fddd806dd50 legacy=0x7fddd8070210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:54.660 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:54.658+0000 7fddfcf02640 10 monclient: shutdown 2024-03-25T18:17:54.660 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:54.658+0000 7fddfcf02640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:54.660 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:54.658+0000 7fddfcf02640 1 -- 172.21.15.97:0/3179888891 >> v1:172.21.15.97:6789/0 conn(0x7fddf80b1730 legacy=0x7fddf81d7ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:54.661 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:54.658+0000 7fddfcf02640 1 -- 172.21.15.97:0/3179888891 shutdown_connections 2024-03-25T18:17:54.661 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:54.658+0000 7fddfcf02640 1 -- 172.21.15.97:0/3179888891 >> 172.21.15.97:0/3179888891 conn(0x7fddf80617a0 msgr2=0x7fddf80b7c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:54.661 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:54.658+0000 7fddfcf02640 1 -- 172.21.15.97:0/3179888891 shutdown_connections 2024-03-25T18:17:54.661 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:54.658+0000 7fddfcf02640 1 -- 172.21.15.97:0/3179888891 wait complete. 2024-03-25T18:17:54.661 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:17:54.659+0000 7fddfcf02640 1 librados: shutdown 2024-03-25T18:17:54.671 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:18:01.478 DEBUG:teuthology.orchestra.run.smithi097:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:18:01.588 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.586+0000 7f4725486640 10 monclient: get_monmap_and_config 2024-03-25T18:18:01.588 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.586+0000 7f4725486640 10 monclient: build_initial_monmap 2024-03-25T18:18:01.588 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.586+0000 7f4725486640 10 monclient: monmap: 2024-03-25T18:18:01.588 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:18:01.588 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:18:01.588 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:18:01.588157+0000 2024-03-25T18:18:01.588 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:18:01.588157+0000 2024-03-25T18:18:01.589 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:01.589 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:18:01.589 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:18:01.589 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T18:18:01.589 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-b 2024-03-25T18:18:01.589 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:18:01.590 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.587+0000 7f4725486640 1 Processor -- start 2024-03-25T18:18:01.590 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.587+0000 7f4725486640 1 -- start start 2024-03-25T18:18:01.591 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.587+0000 7f4725486640 10 monclient: init 2024-03-25T18:18:01.591 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.588+0000 7f4725486640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:01.591 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.588+0000 7f4725486640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:18:01.591 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.588+0000 7f4725486640 10 monclient(hunting): picked mon.noname-c con 0x7f472006fb30 addr v1:172.21.15.98:6789/0 2024-03-25T18:18:01.591 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.588+0000 7f4725486640 10 monclient(hunting): picked mon.noname-a con 0x7f472013c0b0 addr v1:172.21.15.97:6789/0 2024-03-25T18:18:01.591 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.588+0000 7f4725486640 10 monclient(hunting): picked mon.noname-b con 0x7f472006ff10 addr v1:172.21.15.97:6790/0 2024-03-25T18:18:01.591 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.588+0000 7f4725486640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4720071450 con 0x7f472013c0b0 2024-03-25T18:18:01.591 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.588+0000 7f4725486640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4720071730 con 0x7f472006fb30 2024-03-25T18:18:01.591 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.588+0000 7f4725486640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47200730f0 con 0x7f472006ff10 2024-03-25T18:18:01.591 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.588+0000 7f4725486640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:01.591 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.588+0000 7f4725486640 10 monclient(hunting): authenticate will time out at 7693.864769s 2024-03-25T18:18:01.600 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.599+0000 7f471e7fc640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f472013c0b0 0x7f4720071b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:57000/0 (socket says 172.21.15.97:57000) 2024-03-25T18:18:01.600 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.599+0000 7f471e7fc640 1 -- 172.21.15.97:0/2948668794 learned_addr learned my addr 172.21.15.97:0/2948668794 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:18:01.610 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.609+0000 7f471dffb640 1 -- 172.21.15.97:0/2948668794 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2267863615 0 0) 0x7f4720071730 con 0x7f472006fb30 2024-03-25T18:18:01.610 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.609+0000 7f471dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:18:01.610 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.609+0000 7f471dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:01.611 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.609+0000 7f471dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:01.611 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.609+0000 7f471dffb640 10 monclient(hunting): my global_id is 4291 2024-03-25T18:18:01.611 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.609+0000 7f471dffb640 1 -- 172.21.15.97:0/2948668794 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4708003440 con 0x7f472006fb30 2024-03-25T18:18:01.617 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.616+0000 7f471dffb640 1 -- 172.21.15.97:0/2948668794 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2248614840 0 0) 0x7f4720071450 con 0x7f472013c0b0 2024-03-25T18:18:01.617 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.616+0000 7f471dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:18:01.617 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.616+0000 7f471dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:01.617 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.616+0000 7f471dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:01.617 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.616+0000 7f471dffb640 10 monclient(hunting): my global_id is 4296 2024-03-25T18:18:01.617 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.616+0000 7f471dffb640 1 -- 172.21.15.97:0/2948668794 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4720071730 con 0x7f472013c0b0 2024-03-25T18:18:01.622 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.620+0000 7f471dffb640 1 -- 172.21.15.97:0/2948668794 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 793332429 0 0) 0x7f47200730f0 con 0x7f472006ff10 2024-03-25T18:18:01.622 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.620+0000 7f471dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:18:01.622 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.620+0000 7f471dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:01.622 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.620+0000 7f471dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:01.622 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.620+0000 7f471dffb640 10 monclient(hunting): my global_id is 4286 2024-03-25T18:18:01.622 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.620+0000 7f471dffb640 1 -- 172.21.15.97:0/2948668794 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4720071450 con 0x7f472006ff10 2024-03-25T18:18:01.622 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.620+0000 7f471dffb640 1 -- 172.21.15.97:0/2948668794 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2504661482 0 0) 0x7f4708003440 con 0x7f472006fb30 2024-03-25T18:18:01.622 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.620+0000 7f471dffb640 1 -- 172.21.15.97:0/2948668794 >> v1:172.21.15.97:6790/0 conn(0x7f472006ff10 legacy=0x7f472014a050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:01.623 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.620+0000 7f471dffb640 1 -- 172.21.15.97:0/2948668794 >> v1:172.21.15.97:6789/0 conn(0x7f472013c0b0 legacy=0x7f4720071b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:01.623 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.620+0000 7f471dffb640 10 monclient: _finish_hunting 0 2024-03-25T18:18:01.623 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.620+0000 7f471dffb640 1 monclient: found mon.noname-c 2024-03-25T18:18:01.623 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.620+0000 7f471dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:01.623 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.620+0000 7f471dffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.98:6789/0 2024-03-25T18:18:01.623 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.620+0000 7f471dffb640 1 -- 172.21.15.97:0/2948668794 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f472006ef90 con 0x7f472006fb30 2024-03-25T18:18:01.623 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.620+0000 7f471dffb640 10 monclient: _finish_auth 0 2024-03-25T18:18:01.623 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.620+0000 7f471dffb640 10 monclient: _check_auth_tickets 2024-03-25T18:18:01.623 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.620+0000 7f471dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:01.623 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.620+0000 7f471dffb640 1 -- 172.21.15.97:0/2948668794 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f4710001690 con 0x7f472006fb30 2024-03-25T18:18:01.623 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.620+0000 7f471dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:01.623 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.620+0000 7f471dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:18:01.623 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.620+0000 7f471dffb640 10 monclient: dump: 2024-03-25T18:18:01.623 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:18:01.623 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:18:01.624 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:18:01.624 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:18:01.624 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:01.624 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:18:01.624 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:18:01.624 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:18:01.624 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:18:01.624 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:18:01.624 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.620+0000 7f4725486640 5 monclient: authenticate success, global_id 4291 2024-03-25T18:18:01.624 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.620+0000 7f4725486640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:18:01.629 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.628+0000 7f471dffb640 1 -- 172.21.15.97:0/2948668794 <== mon.1 v1:172.21.15.98:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f47100017f0 con 0x7f472006fb30 2024-03-25T18:18:01.629 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.628+0000 7f471dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:01.629 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.628+0000 7f471dffb640 1 -- 172.21.15.97:0/2948668794 <== mon.1 v1:172.21.15.98:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f4710001c00 con 0x7f472006fb30 2024-03-25T18:18:01.629 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.628+0000 7f471dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:01.629 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.628+0000 7f471dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:18:01.629 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.628+0000 7f471dffb640 10 monclient: dump: 2024-03-25T18:18:01.629 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:18:01.630 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:18:01.630 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:18:01.630 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:18:01.630 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:01.630 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:18:01.630 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:18:01.630 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:18:01.630 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:18:01.630 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:18:01.630 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.628+0000 7f4725486640 10 monclient: get_monmap_and_config success 2024-03-25T18:18:01.630 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.628+0000 7f4725486640 4 set_mon_vals no callback set 2024-03-25T18:18:01.631 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.629+0000 7f4725486640 10 monclient: shutdown 2024-03-25T18:18:01.632 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.629+0000 7f4725486640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:01.632 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.629+0000 7f4725486640 1 -- 172.21.15.97:0/2948668794 >> v1:172.21.15.98:6789/0 conn(0x7f472006fb30 legacy=0x7f472013b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:01.632 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.629+0000 7f4725486640 1 -- 172.21.15.97:0/2948668794 shutdown_connections 2024-03-25T18:18:01.632 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.629+0000 7f4725486640 1 -- 172.21.15.97:0/2948668794 >> 172.21.15.97:0/2948668794 conn(0x7f47200b11e0 msgr2=0x7f47200b15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:01.632 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.629+0000 7f4725486640 1 -- 172.21.15.97:0/2948668794 shutdown_connections 2024-03-25T18:18:01.632 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.629+0000 7f4725486640 1 -- 172.21.15.97:0/2948668794 wait complete. 2024-03-25T18:18:01.632 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.630+0000 7f4725486640 10 monclient: build_initial_monmap 2024-03-25T18:18:01.632 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.630+0000 7f4725486640 10 monclient: monmap: 2024-03-25T18:18:01.632 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:18:01.632 INFO:teuthology.misc.health.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:18:01.632 INFO:teuthology.misc.health.smithi097.stderr:last_changed 0.000000 2024-03-25T18:18:01.632 INFO:teuthology.misc.health.smithi097.stderr:created 0.000000 2024-03-25T18:18:01.632 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:01.633 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 1 2024-03-25T18:18:01.633 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:18:01.633 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T18:18:01.633 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-c 2024-03-25T18:18:01.633 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:18:01.633 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.630+0000 7f4725486640 1 librados: starting msgr at 2024-03-25T18:18:01.633 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.630+0000 7f4725486640 1 librados: starting objecter 2024-03-25T18:18:01.633 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.630+0000 7f4725486640 1 Processor -- start 2024-03-25T18:18:01.633 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.631+0000 7f4725486640 1 -- start start 2024-03-25T18:18:01.633 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.631+0000 7f4725486640 1 librados: setting wanted keys 2024-03-25T18:18:01.633 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.631+0000 7f4725486640 1 librados: calling monclient init 2024-03-25T18:18:01.633 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.631+0000 7f4725486640 10 monclient: init 2024-03-25T18:18:01.634 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.631+0000 7f4725486640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:01.634 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.631+0000 7f4725486640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:18:01.635 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.631+0000 7f4725486640 10 monclient(hunting): picked mon.noname-a con 0x7f472006fb30 addr v1:172.21.15.97:6789/0 2024-03-25T18:18:01.635 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.631+0000 7f4725486640 10 monclient(hunting): picked mon.noname-b con 0x7f472006ff10 addr v1:172.21.15.98:6789/0 2024-03-25T18:18:01.635 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.632+0000 7f4725486640 10 monclient(hunting): picked mon.noname-c con 0x7f472013c0b0 addr v1:172.21.15.97:6790/0 2024-03-25T18:18:01.635 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.632+0000 7f4725486640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47201db8c0 con 0x7f472006fb30 2024-03-25T18:18:01.635 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.632+0000 7f4725486640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47201db310 con 0x7f472006ff10 2024-03-25T18:18:01.635 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.632+0000 7f4725486640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47201db4a0 con 0x7f472013c0b0 2024-03-25T18:18:01.635 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.632+0000 7f4725486640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:01.635 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.632+0000 7f4725486640 10 monclient(hunting): authenticate will time out at 7693.908185s 2024-03-25T18:18:01.639 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.638+0000 7f471e7fc640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f472006ff10 0x7f4720137990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.97:38568/0 (socket says 172.21.15.97:38568) 2024-03-25T18:18:01.639 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.638+0000 7f471e7fc640 1 -- 172.21.15.97:0/3073788135 learned_addr learned my addr 172.21.15.97:0/3073788135 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:18:01.643 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.642+0000 7f47077fe640 10 client.?.objecter ms_handle_connect 0x7f472006ff10 2024-03-25T18:18:01.643 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.642+0000 7f47077fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:01.645 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.643+0000 7f47077fe640 10 client.?.objecter ms_handle_connect 0x7f472006fb30 2024-03-25T18:18:01.645 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.643+0000 7f47077fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:01.652 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.651+0000 7f47077fe640 1 -- 172.21.15.97:0/3073788135 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 851404649 0 0) 0x7f47201db310 con 0x7f472006ff10 2024-03-25T18:18:01.652 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.651+0000 7f47077fe640 10 monclient(hunting): _negotiate 2024-03-25T18:18:01.653 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.651+0000 7f47077fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:01.653 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.651+0000 7f47077fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:01.653 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.651+0000 7f47077fe640 10 monclient(hunting): my global_id is 4294 2024-03-25T18:18:01.653 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.651+0000 7f47077fe640 1 -- 172.21.15.97:0/3073788135 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f470c004eb0 con 0x7f472006ff10 2024-03-25T18:18:01.658 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.657+0000 7f47077fe640 10 client.?.objecter ms_handle_connect 0x7f472013c0b0 2024-03-25T18:18:01.658 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.657+0000 7f47077fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:01.660 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.659+0000 7f47077fe640 1 -- 172.21.15.97:0/3073788135 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3115628237 0 0) 0x7f47201db8c0 con 0x7f472006fb30 2024-03-25T18:18:01.660 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.659+0000 7f47077fe640 10 monclient(hunting): _negotiate 2024-03-25T18:18:01.660 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.659+0000 7f47077fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:01.660 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.659+0000 7f47077fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:01.660 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.659+0000 7f47077fe640 10 monclient(hunting): my global_id is 4299 2024-03-25T18:18:01.660 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.659+0000 7f47077fe640 1 -- 172.21.15.97:0/3073788135 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47201db310 con 0x7f472006fb30 2024-03-25T18:18:01.662 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.661+0000 7f47077fe640 1 -- 172.21.15.97:0/3073788135 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 978171136 0 0) 0x7f470c004eb0 con 0x7f472006ff10 2024-03-25T18:18:01.662 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.661+0000 7f47077fe640 1 -- 172.21.15.97:0/3073788135 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f47201db8c0 con 0x7f472006ff10 2024-03-25T18:18:01.663 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.661+0000 7f47077fe640 1 -- 172.21.15.97:0/3073788135 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f4714001970 con 0x7f472006ff10 2024-03-25T18:18:01.663 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.661+0000 7f47077fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:01.663 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.661+0000 7f47077fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:18:01.663 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.661+0000 7f47077fe640 10 monclient(hunting): dump: 2024-03-25T18:18:01.663 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:18:01.663 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:18:01.663 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:18:01.663 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:18:01.663 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:01.663 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:18:01.663 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:18:01.663 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:18:01.663 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:18:01.663 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:18:01.664 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.661+0000 7f47077fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:18:01.675 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.674+0000 7f47077fe640 1 -- 172.21.15.97:0/3073788135 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3937550726 0 0) 0x7f47201db4a0 con 0x7f472013c0b0 2024-03-25T18:18:01.675 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.674+0000 7f47077fe640 10 monclient(hunting): _negotiate 2024-03-25T18:18:01.675 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.674+0000 7f47077fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:01.675 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.674+0000 7f47077fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:01.675 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.674+0000 7f47077fe640 10 monclient(hunting): my global_id is 4289 2024-03-25T18:18:01.675 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.674+0000 7f47077fe640 1 -- 172.21.15.97:0/3073788135 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f470c004eb0 con 0x7f472013c0b0 2024-03-25T18:18:01.676 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.674+0000 7f47077fe640 1 -- 172.21.15.97:0/3073788135 <== mon.1 v1:172.21.15.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2424791100 0 0) 0x7f47201db8c0 con 0x7f472006ff10 2024-03-25T18:18:01.676 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.675+0000 7f47077fe640 1 -- 172.21.15.97:0/3073788135 >> v1:172.21.15.97:6790/0 conn(0x7f472013c0b0 legacy=0x7f47200ac1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:01.677 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.675+0000 7f47077fe640 1 -- 172.21.15.97:0/3073788135 >> v1:172.21.15.97:6789/0 conn(0x7f472006fb30 legacy=0x7f47201dac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:01.677 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.675+0000 7f47077fe640 10 monclient: _finish_hunting 0 2024-03-25T18:18:01.677 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.675+0000 7f47077fe640 1 monclient: found mon.b 2024-03-25T18:18:01.677 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.675+0000 7f47077fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:01.677 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.675+0000 7f47077fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:18:01.677 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.675+0000 7f47077fe640 1 -- 172.21.15.97:0/3073788135 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4720138bd0 con 0x7f472006ff10 2024-03-25T18:18:01.677 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.675+0000 7f47077fe640 10 monclient: _finish_auth 0 2024-03-25T18:18:01.677 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.675+0000 7f47077fe640 10 monclient: _check_auth_tickets 2024-03-25T18:18:01.677 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.675+0000 7f47077fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:01.677 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.675+0000 7f4725486640 5 monclient: authenticate success, global_id 4294 2024-03-25T18:18:01.677 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.675+0000 7f4725486640 10 monclient: _renew_subs 2024-03-25T18:18:01.677 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.675+0000 7f4725486640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:18:01.677 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.675+0000 7f4725486640 1 -- 172.21.15.97:0/3073788135 --> v1:172.21.15.98:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4720138e20 con 0x7f472006ff10 2024-03-25T18:18:01.677 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.675+0000 7f4725486640 10 client.4294.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:18:01.677 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.675+0000 7f4725486640 10 monclient: _renew_subs 2024-03-25T18:18:01.678 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.675+0000 7f4725486640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:18:01.678 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.675+0000 7f4725486640 1 -- 172.21.15.97:0/3073788135 --> v1:172.21.15.98:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4720138280 con 0x7f472006ff10 2024-03-25T18:18:01.678 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.675+0000 7f4725486640 1 librados: init done 2024-03-25T18:18:01.678 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.676+0000 7f4725486640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:01.678 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.676+0000 7f4725486640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:01.678 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.676+0000 7f4725486640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:18:01.678 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.676+0000 7f4725486640 1 -- 172.21.15.97:0/3073788135 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47201db630 con 0x7f472006ff10 2024-03-25T18:18:01.943 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.942+0000 7f471e7fc640 1 -- 172.21.15.97:0/3073788135 >> v1:172.21.15.98:6789/0 conn(0x7f472006ff10 legacy=0x7f4720137990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-03-25T18:18:01.943 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.942+0000 7f47077fe640 1 -- 172.21.15.97:0/3073788135 <== mon.1 v1:172.21.15.98:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f47140047e0 con 0x7f472006ff10 2024-03-25T18:18:01.944 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.942+0000 7f47077fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:01.944 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.942+0000 7f471e7fc640 1 -- 172.21.15.97:0/3073788135 >> v1:172.21.15.98:6789/0 conn(0x7f472006ff10 legacy=0x7f4720137990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T18:18:01.944 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.942+0000 7f471e7fc640 1 --1- 172.21.15.97:0/3073788135 >> v1:172.21.15.98:6789/0 conn(0x7f472006ff10 0x7f4720137990 :-1 s=OPENED pgs=8799 cs=1 l=1).handle_message read tag failed 2024-03-25T18:18:01.944 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.942+0000 7f471e7fc640 1 --1- 172.21.15.97:0/3073788135 >> v1:172.21.15.98:6789/0 conn(0x7f472006ff10 0x7f4720137990 :-1 s=OPENED pgs=8799 cs=1 l=1).fault on lossy channel, failing 2024-03-25T18:18:01.944 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.942+0000 7f47077fe640 1 -- 172.21.15.97:0/3073788135 <== mon.1 v1:172.21.15.98:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f4714004af0 con 0x7f472006ff10 2024-03-25T18:18:01.944 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.942+0000 7f47077fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:01.944 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.942+0000 7f47077fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:18:01.944 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.942+0000 7f47077fe640 10 monclient: dump: 2024-03-25T18:18:01.944 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:18:01.944 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:18:01.944 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:18:01.945 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:18:01.945 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:01.945 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:18:01.945 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:18:01.945 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:18:01.945 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:18:01.945 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:18:01.945 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.942+0000 7f47077fe640 10 monclient: ms_handle_reset current mon v1:172.21.15.98:6789/0 2024-03-25T18:18:01.945 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.942+0000 7f47077fe640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:01.945 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.942+0000 7f47077fe640 1 -- 172.21.15.97:0/3073788135 >> v1:172.21.15.98:6789/0 conn(0x7f472006ff10 legacy=0x7f4720137990 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T18:18:01.945 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.942+0000 7f471d7fa640 4 set_mon_vals no callback set 2024-03-25T18:18:01.946 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.943+0000 7f47077fe640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:18:01.946 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.943+0000 7f47077fe640 10 monclient(hunting): picked mon.c con 0x7f472013c0b0 addr v1:172.21.15.97:6790/0 2024-03-25T18:18:01.946 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.943+0000 7f47077fe640 10 monclient(hunting): picked mon.a con 0x7f472006fb30 addr v1:172.21.15.97:6789/0 2024-03-25T18:18:01.946 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.943+0000 7f47077fe640 10 monclient(hunting): picked mon.b con 0x7f470c00da30 addr v1:172.21.15.98:6789/0 2024-03-25T18:18:01.947 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.943+0000 7f47077fe640 1 -- 172.21.15.97:0/3073788135 --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f47201db8c0 con 0x7f472006fb30 2024-03-25T18:18:01.947 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.943+0000 7f47077fe640 1 -- 172.21.15.97:0/3073788135 --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f47201db4a0 con 0x7f470c00da30 2024-03-25T18:18:01.947 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.943+0000 7f47077fe640 1 -- 172.21.15.97:0/3073788135 --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f470c013a90 con 0x7f472013c0b0 2024-03-25T18:18:01.947 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.943+0000 7f47077fe640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:01.963 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.962+0000 7f47077fe640 10 client.4294.objecter ms_handle_connect 0x7f472006fb30 2024-03-25T18:18:01.963 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.962+0000 7f47077fe640 10 client.4294.objecter resend_mon_ops 2024-03-25T18:18:01.971 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.969+0000 7f47077fe640 1 -- 172.21.15.97:0/3073788135 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 24684056 0 0) 0x7f47201db8c0 con 0x7f472006fb30 2024-03-25T18:18:01.971 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.969+0000 7f47077fe640 10 monclient(hunting): _negotiate 2024-03-25T18:18:01.971 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.969+0000 7f47077fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:01.971 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.969+0000 7f47077fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-25T18:18:01.971 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.969+0000 7f47077fe640 1 -- 172.21.15.97:0/3073788135 --> v1:172.21.15.97:6789/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f470c0119e0 con 0x7f472006fb30 2024-03-25T18:18:01.975 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.974+0000 7f47077fe640 10 client.4294.objecter ms_handle_connect 0x7f472013c0b0 2024-03-25T18:18:01.975 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.974+0000 7f47077fe640 10 client.4294.objecter resend_mon_ops 2024-03-25T18:18:01.980 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.979+0000 7f47077fe640 1 -- 172.21.15.97:0/3073788135 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2330357575 0 0) 0x7f470c013a90 con 0x7f472013c0b0 2024-03-25T18:18:01.980 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.979+0000 7f47077fe640 10 monclient(hunting): _negotiate 2024-03-25T18:18:01.980 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.979+0000 7f47077fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:01.980 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.979+0000 7f47077fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-25T18:18:01.980 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.979+0000 7f47077fe640 1 -- 172.21.15.97:0/3073788135 --> v1:172.21.15.97:6790/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f47201db8c0 con 0x7f472013c0b0 2024-03-25T18:18:01.984 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.982+0000 7f47077fe640 10 client.4294.objecter ms_handle_connect 0x7f470c00da30 2024-03-25T18:18:01.984 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.982+0000 7f47077fe640 10 client.4294.objecter resend_mon_ops 2024-03-25T18:18:01.988 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.987+0000 7f47077fe640 1 -- 172.21.15.97:0/3073788135 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 990715817 0 0) 0x7f47201db4a0 con 0x7f470c00da30 2024-03-25T18:18:01.988 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.987+0000 7f47077fe640 10 monclient(hunting): _negotiate 2024-03-25T18:18:01.989 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.987+0000 7f47077fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:01.989 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.987+0000 7f47077fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-25T18:18:01.989 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.987+0000 7f47077fe640 1 -- 172.21.15.97:0/3073788135 --> v1:172.21.15.98:6789/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f470c013a90 con 0x7f470c00da30 2024-03-25T18:18:01.998 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.996+0000 7f47077fe640 1 -- 172.21.15.97:0/3073788135 <== mon.2 v1:172.21.15.97:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 233+0+0 (unknown 2724457281 0 0) 0x7f47201db8c0 con 0x7f472013c0b0 2024-03-25T18:18:01.998 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.996+0000 7f47077fe640 1 -- 172.21.15.97:0/3073788135 >> v1:172.21.15.98:6789/0 conn(0x7f470c00da30 legacy=0x7f470c00fe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:01.998 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.996+0000 7f47077fe640 1 -- 172.21.15.97:0/3073788135 >> v1:172.21.15.97:6789/0 conn(0x7f472006fb30 legacy=0x7f470c00c310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:01.998 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.996+0000 7f47077fe640 10 monclient: _finish_hunting 0 2024-03-25T18:18:01.998 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.996+0000 7f47077fe640 1 monclient: found mon.c 2024-03-25T18:18:01.998 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.996+0000 7f47077fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:01.998 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.996+0000 7f47077fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.97:6790/0 2024-03-25T18:18:01.999 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.996+0000 7f47077fe640 1 -- 172.21.15.97:0/3073788135 --> v1:172.21.15.97:6790/0 -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+,osdmap=0}) v3 -- 0x7f470c014e70 con 0x7f472013c0b0 2024-03-25T18:18:01.999 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.996+0000 7f47077fe640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:01.999 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.996+0000 7f47077fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.97:6790/0 2024-03-25T18:18:01.999 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.996+0000 7f47077fe640 1 -- 172.21.15.97:0/3073788135 --> v1:172.21.15.97:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47201db4a0 con 0x7f472013c0b0 2024-03-25T18:18:01.999 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.996+0000 7f47077fe640 10 monclient: _finish_auth 0 2024-03-25T18:18:01.999 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.996+0000 7f47077fe640 10 monclient: _check_auth_tickets 2024-03-25T18:18:01.999 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.996+0000 7f47077fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:01.999 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.996+0000 7f47077fe640 1 -- 172.21.15.97:0/3073788135 <== mon.2 v1:172.21.15.97:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f47100015d0 con 0x7f472013c0b0 2024-03-25T18:18:01.999 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.996+0000 7f47077fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:01.999 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.996+0000 7f47077fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:18:01.999 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:01.996+0000 7f47077fe640 10 monclient: dump: 2024-03-25T18:18:01.999 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:18:01.999 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:18:01.999 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:18:02.000 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:18:02.000 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:02.000 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:18:02.000 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:18:02.000 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:18:02.000 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:18:02.000 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:18:02.317 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:02.315+0000 7f47077fe640 1 -- 172.21.15.97:0/3073788135 <== mon.2 v1:172.21.15.97:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4710001960 con 0x7f472013c0b0 2024-03-25T18:18:02.317 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:02.315+0000 7f47077fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:02.317 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:02.315+0000 7f471cff9640 4 set_mon_vals no callback set 2024-03-25T18:18:02.317 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:02.316+0000 7f47077fe640 1 -- 172.21.15.97:0/3073788135 <== mon.2 v1:172.21.15.97:6790/0 5 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2377622137 0 0) 0x7f4710019460 con 0x7f472013c0b0 2024-03-25T18:18:02.318 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:02.317+0000 7f471e7fc640 10 monclient: get_auth_request con 0x7f472006ff10 auth_method 0 2024-03-25T18:18:02.318 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:02.317+0000 7f47077fe640 10 client.4294.objecter ms_handle_connect 0x7f472006ff10 2024-03-25T18:18:02.324 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:02.322+0000 7f47077fe640 1 -- 172.21.15.97:0/3073788135 <== mon.2 v1:172.21.15.97:6790/0 6 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 430711918 0 0) 0x7f4710052d10 con 0x7f472013c0b0 2024-03-25T18:18:02.324 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:02.322+0000 7f47077fe640 10 client.4294.objecter ms_dispatch 0x7f4720000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:18:02.324 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:02.322+0000 7f47077fe640 3 client.4294.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:18:02.324 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:02.322+0000 7f47077fe640 3 client.4294.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:18:02.324 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:02.322+0000 7f47077fe640 20 client.4294.objecter dump_active .. 0 homeless 2024-03-25T18:18:02.324 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:02.323+0000 7f47077fe640 1 -- 172.21.15.97:0/3073788135 <== mon.2 v1:172.21.15.97:6790/0 7 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f471004e650 con 0x7f472013c0b0 2024-03-25T18:18:02.325 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:02.323+0000 7f47077fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:02.325 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:02.323+0000 7f47077fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:18:02.601 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:02.599+0000 7f4725486640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:02.601 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:02.599+0000 7f4725486640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:02.601 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:02.599+0000 7f4725486640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.97:6790/0 2024-03-25T18:18:02.601 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:02.599+0000 7f4725486640 1 -- 172.21.15.97:0/3073788135 --> v1:172.21.15.97:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f47200abec0 con 0x7f472013c0b0 2024-03-25T18:18:02.638 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:02.637+0000 7f47077fe640 1 -- 172.21.15.97:0/3073788135 <== mon.2 v1:172.21.15.97:6790/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2169427412) 0x7f47200abec0 con 0x7f472013c0b0 2024-03-25T18:18:02.638 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:02.637+0000 7f47077fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:02.638 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:02.637+0000 7f47077fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:18:02.639 INFO:teuthology.misc.health.smithi097.stdout: 2024-03-25T18:18:02.639 INFO:teuthology.misc.health.smithi097.stdout:{"epoch":10,"fsid":"4c3d472c-79b8-486b-9e3d-2738d4306754","created":"2024-03-25T18:16:14.737309+0000","modified":"2024-03-25T18:16:20.357492+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:20.357492+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cb5152b9-262f-45c3-94b8-18d0e81df558","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6daf8793-0838-473d-a7d6-9a0d2894c75d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"136f1493-3b9c-4f43-a2e7-641482d69dae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"397d538c-4f64-4daf-9e8b-662d09f91327","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4f1df08-663f-4bda-93f8-3878ab7c24b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c4b18d70-a3ce-4b15-9b70-eae5525bd761","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8dd4033a-de0e-4f8a-96dc-59f3c1f825b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0077722e-ffa6-4a3d-9891-929d9869b286","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:18:02.641 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:02.639+0000 7f4725486640 10 librados: watch_flush enter 2024-03-25T18:18:02.641 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:02.639+0000 7f4725486640 10 librados: watch_flush exit 2024-03-25T18:18:02.641 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:02.639+0000 7f4725486640 20 client.4294.objecter shutdown clearing up homeless session... 2024-03-25T18:18:02.641 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:02.639+0000 7f4725486640 10 client.4294.objecter successfully canceled tick 2024-03-25T18:18:02.642 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:02.639+0000 7f4725486640 1 -- 172.21.15.97:0/3073788135 >> v1:172.21.15.98:6800/800387573 conn(0x7f472006ff10 legacy=0x7f470c07bc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:02.642 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:02.639+0000 7f4725486640 10 monclient: shutdown 2024-03-25T18:18:02.642 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:02.639+0000 7f4725486640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:02.642 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:02.639+0000 7f4725486640 1 -- 172.21.15.97:0/3073788135 >> v1:172.21.15.97:6790/0 conn(0x7f472013c0b0 legacy=0x7f470c0064f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:02.642 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:02.639+0000 7f4725486640 1 -- 172.21.15.97:0/3073788135 shutdown_connections 2024-03-25T18:18:02.642 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:02.639+0000 7f4725486640 1 -- 172.21.15.97:0/3073788135 >> 172.21.15.97:0/3073788135 conn(0x7f47200b11e0 msgr2=0x7f4720074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:02.642 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:02.639+0000 7f4725486640 1 -- 172.21.15.97:0/3073788135 shutdown_connections 2024-03-25T18:18:02.642 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:02.639+0000 7f4725486640 1 -- 172.21.15.97:0/3073788135 wait complete. 2024-03-25T18:18:02.642 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:02.639+0000 7f4725486640 1 librados: shutdown 2024-03-25T18:18:02.653 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:18:09.460 DEBUG:teuthology.orchestra.run.smithi097:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:18:09.572 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.571+0000 7f07aa3e6640 10 monclient: get_monmap_and_config 2024-03-25T18:18:09.573 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.571+0000 7f07aa3e6640 10 monclient: build_initial_monmap 2024-03-25T18:18:09.573 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.571+0000 7f07aa3e6640 10 monclient: monmap: 2024-03-25T18:18:09.573 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:18:09.573 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:18:09.573 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:18:09.572600+0000 2024-03-25T18:18:09.573 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:18:09.572600+0000 2024-03-25T18:18:09.573 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:09.573 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:18:09.573 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:18:09.573 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T18:18:09.574 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-b 2024-03-25T18:18:09.574 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:18:09.575 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.572+0000 7f07aa3e6640 1 Processor -- start 2024-03-25T18:18:09.575 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.572+0000 7f07aa3e6640 1 -- start start 2024-03-25T18:18:09.575 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.572+0000 7f07aa3e6640 10 monclient: init 2024-03-25T18:18:09.575 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.572+0000 7f07aa3e6640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:09.575 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.572+0000 7f07aa3e6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:18:09.575 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.573+0000 7f07aa3e6640 10 monclient(hunting): picked mon.noname-b con 0x7f07a40a98f0 addr v1:172.21.15.97:6790/0 2024-03-25T18:18:09.575 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.573+0000 7f07aa3e6640 10 monclient(hunting): picked mon.noname-c con 0x7f07a413f710 addr v1:172.21.15.98:6789/0 2024-03-25T18:18:09.575 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.573+0000 7f07aa3e6640 10 monclient(hunting): picked mon.noname-a con 0x7f07a41430c0 addr v1:172.21.15.97:6789/0 2024-03-25T18:18:09.576 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.573+0000 7f07aa3e6640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07a4142520 con 0x7f07a41430c0 2024-03-25T18:18:09.576 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.573+0000 7f07aa3e6640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07a40b28a0 con 0x7f07a413f710 2024-03-25T18:18:09.576 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.573+0000 7f07aa3e6640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07a40b2a30 con 0x7f07a40a98f0 2024-03-25T18:18:09.576 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.573+0000 7f07aa3e6640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:09.576 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.573+0000 7f07aa3e6640 10 monclient(hunting): authenticate will time out at 7701.849051s 2024-03-25T18:18:09.583 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.581+0000 7f07a37fe640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f07a413f710 0x7f07a41429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.97:43618/0 (socket says 172.21.15.97:43618) 2024-03-25T18:18:09.583 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.582+0000 7f07a37fe640 1 -- 172.21.15.97:0/946659064 learned_addr learned my addr 172.21.15.97:0/946659064 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:18:09.593 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.592+0000 7f07a2ffd640 1 -- 172.21.15.97:0/946659064 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 193529924 0 0) 0x7f07a40b28a0 con 0x7f07a413f710 2024-03-25T18:18:09.593 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.592+0000 7f07a2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:18:09.594 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.592+0000 7f07a2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:09.594 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.592+0000 7f07a2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:09.594 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.592+0000 7f07a2ffd640 10 monclient(hunting): my global_id is 4297 2024-03-25T18:18:09.594 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.592+0000 7f07a2ffd640 1 -- 172.21.15.97:0/946659064 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f078c003520 con 0x7f07a413f710 2024-03-25T18:18:09.598 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.596+0000 7f07a2ffd640 1 -- 172.21.15.97:0/946659064 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2638287767 0 0) 0x7f078c003520 con 0x7f07a413f710 2024-03-25T18:18:09.598 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.596+0000 7f07a2ffd640 1 -- 172.21.15.97:0/946659064 >> v1:172.21.15.97:6790/0 conn(0x7f07a40a98f0 legacy=0x7f07a40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:09.598 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.596+0000 7f07a2ffd640 1 -- 172.21.15.97:0/946659064 >> v1:172.21.15.97:6789/0 conn(0x7f07a41430c0 legacy=0x7f07a414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:09.598 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.596+0000 7f07a2ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:18:09.598 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.596+0000 7f07a2ffd640 1 monclient: found mon.noname-c 2024-03-25T18:18:09.598 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.596+0000 7f07a2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:09.598 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.596+0000 7f07a2ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.98:6789/0 2024-03-25T18:18:09.599 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.596+0000 7f07a2ffd640 1 -- 172.21.15.97:0/946659064 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07a40b44c0 con 0x7f07a413f710 2024-03-25T18:18:09.599 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.596+0000 7f07a2ffd640 10 monclient: _finish_auth 0 2024-03-25T18:18:09.599 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.596+0000 7f07a2ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:18:09.599 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.596+0000 7f07a2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:09.599 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.596+0000 7f07a2ffd640 1 -- 172.21.15.97:0/946659064 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f0798001690 con 0x7f07a413f710 2024-03-25T18:18:09.599 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.596+0000 7f07aa3e6640 5 monclient: authenticate success, global_id 4297 2024-03-25T18:18:09.599 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.596+0000 7f07a2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:09.599 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.596+0000 7f07a2ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:18:09.599 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.596+0000 7f07a2ffd640 10 monclient: dump: 2024-03-25T18:18:09.599 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:18:09.599 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:18:09.600 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:18:09.600 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:18:09.600 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:09.600 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:18:09.600 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:18:09.600 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:18:09.600 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:18:09.600 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:18:09.600 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.596+0000 7f07aa3e6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:18:09.607 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.606+0000 7f07a2ffd640 1 -- 172.21.15.97:0/946659064 <== mon.1 v1:172.21.15.98:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f07980017f0 con 0x7f07a413f710 2024-03-25T18:18:09.607 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.606+0000 7f07a2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:09.607 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.606+0000 7f07a2ffd640 1 -- 172.21.15.97:0/946659064 <== mon.1 v1:172.21.15.98:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f0798001c00 con 0x7f07a413f710 2024-03-25T18:18:09.607 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.606+0000 7f07aa3e6640 10 monclient: get_monmap_and_config success 2024-03-25T18:18:09.607 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.606+0000 7f07aa3e6640 4 set_mon_vals no callback set 2024-03-25T18:18:09.607 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.606+0000 7f07a2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:09.607 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.606+0000 7f07a2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:18:09.608 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.606+0000 7f07a2ffd640 10 monclient: dump: 2024-03-25T18:18:09.608 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:18:09.608 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:18:09.608 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:18:09.608 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:18:09.608 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:09.608 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:18:09.608 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:18:09.608 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:18:09.608 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:18:09.608 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:18:09.610 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.607+0000 7f07aa3e6640 10 monclient: shutdown 2024-03-25T18:18:09.611 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.607+0000 7f07aa3e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:09.611 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.607+0000 7f07aa3e6640 1 -- 172.21.15.97:0/946659064 >> v1:172.21.15.98:6789/0 conn(0x7f07a413f710 legacy=0x7f07a41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:09.611 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.607+0000 7f07aa3e6640 1 -- 172.21.15.97:0/946659064 shutdown_connections 2024-03-25T18:18:09.611 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.607+0000 7f07aa3e6640 1 -- 172.21.15.97:0/946659064 >> 172.21.15.97:0/946659064 conn(0x7f07a40b7ea0 msgr2=0x7f07a40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:09.611 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.607+0000 7f07aa3e6640 1 -- 172.21.15.97:0/946659064 shutdown_connections 2024-03-25T18:18:09.611 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.607+0000 7f07aa3e6640 1 -- 172.21.15.97:0/946659064 wait complete. 2024-03-25T18:18:09.611 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.607+0000 7f07aa3e6640 10 monclient: build_initial_monmap 2024-03-25T18:18:09.611 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.607+0000 7f07aa3e6640 10 monclient: monmap: 2024-03-25T18:18:09.611 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:18:09.611 INFO:teuthology.misc.health.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:18:09.611 INFO:teuthology.misc.health.smithi097.stderr:last_changed 0.000000 2024-03-25T18:18:09.611 INFO:teuthology.misc.health.smithi097.stderr:created 0.000000 2024-03-25T18:18:09.612 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:09.612 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 1 2024-03-25T18:18:09.612 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:18:09.612 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T18:18:09.612 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-c 2024-03-25T18:18:09.612 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:18:09.612 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.608+0000 7f07aa3e6640 1 librados: starting msgr at 2024-03-25T18:18:09.612 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.608+0000 7f07aa3e6640 1 librados: starting objecter 2024-03-25T18:18:09.612 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.608+0000 7f07aa3e6640 1 Processor -- start 2024-03-25T18:18:09.612 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.608+0000 7f07aa3e6640 1 -- start start 2024-03-25T18:18:09.612 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.608+0000 7f07aa3e6640 1 librados: setting wanted keys 2024-03-25T18:18:09.612 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.608+0000 7f07aa3e6640 1 librados: calling monclient init 2024-03-25T18:18:09.612 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.608+0000 7f07aa3e6640 10 monclient: init 2024-03-25T18:18:09.612 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.608+0000 7f07aa3e6640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:09.613 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.608+0000 7f07aa3e6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:18:09.613 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.609+0000 7f07aa3e6640 10 monclient(hunting): picked mon.noname-c con 0x7f07a40a98f0 addr v1:172.21.15.97:6790/0 2024-03-25T18:18:09.613 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.609+0000 7f07aa3e6640 10 monclient(hunting): picked mon.noname-b con 0x7f07a413f710 addr v1:172.21.15.98:6789/0 2024-03-25T18:18:09.613 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.609+0000 7f07aa3e6640 10 monclient(hunting): picked mon.noname-a con 0x7f07a41430c0 addr v1:172.21.15.97:6789/0 2024-03-25T18:18:09.613 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.609+0000 7f07aa3e6640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07a41370a0 con 0x7f07a41430c0 2024-03-25T18:18:09.613 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.609+0000 7f07aa3e6640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07a4135090 con 0x7f07a413f710 2024-03-25T18:18:09.613 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.609+0000 7f07aa3e6640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07a4135220 con 0x7f07a40a98f0 2024-03-25T18:18:09.613 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.609+0000 7f07aa3e6640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:09.613 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.609+0000 7f07aa3e6640 10 monclient(hunting): authenticate will time out at 7701.885077s 2024-03-25T18:18:09.623 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.622+0000 7f07a37fe640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f07a413f710 0x7f07a41d41e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.97:43652/0 (socket says 172.21.15.97:43652) 2024-03-25T18:18:09.623 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.622+0000 7f07a37fe640 1 -- 172.21.15.97:0/1087733458 learned_addr learned my addr 172.21.15.97:0/1087733458 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:18:09.626 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.625+0000 7f07a0ff9640 10 client.?.objecter ms_handle_connect 0x7f07a413f710 2024-03-25T18:18:09.627 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.625+0000 7f07a0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:09.628 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.627+0000 7f07a0ff9640 10 client.?.objecter ms_handle_connect 0x7f07a41430c0 2024-03-25T18:18:09.629 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.627+0000 7f07a0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:09.632 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.631+0000 7f07a0ff9640 1 -- 172.21.15.97:0/1087733458 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 694442507 0 0) 0x7f07a4135090 con 0x7f07a413f710 2024-03-25T18:18:09.632 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.631+0000 7f07a0ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:18:09.632 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.631+0000 7f07a0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:09.632 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.631+0000 7f07a0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:09.632 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.631+0000 7f07a0ff9640 10 monclient(hunting): my global_id is 4300 2024-03-25T18:18:09.632 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.631+0000 7f07a0ff9640 1 -- 172.21.15.97:0/1087733458 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0790004eb0 con 0x7f07a413f710 2024-03-25T18:18:09.632 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.631+0000 7f07a0ff9640 10 client.?.objecter ms_handle_connect 0x7f07a40a98f0 2024-03-25T18:18:09.632 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.631+0000 7f07a0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:09.642 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.640+0000 7f07a0ff9640 1 -- 172.21.15.97:0/1087733458 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3914470845 0 0) 0x7f0790004eb0 con 0x7f07a413f710 2024-03-25T18:18:09.642 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.641+0000 7f07a0ff9640 1 -- 172.21.15.97:0/1087733458 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f07a4135090 con 0x7f07a413f710 2024-03-25T18:18:09.642 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.641+0000 7f07a0ff9640 1 -- 172.21.15.97:0/1087733458 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f0798001c00 con 0x7f07a413f710 2024-03-25T18:18:09.642 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.641+0000 7f07a0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:09.642 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.641+0000 7f07a0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:18:09.642 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.641+0000 7f07a0ff9640 10 monclient(hunting): dump: 2024-03-25T18:18:09.642 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:18:09.643 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:18:09.643 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:18:09.643 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:18:09.643 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:09.643 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:18:09.643 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:18:09.643 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:18:09.643 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:18:09.643 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:18:09.643 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.641+0000 7f07a0ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:18:09.647 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.646+0000 7f07a0ff9640 1 -- 172.21.15.97:0/1087733458 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2468565565 0 0) 0x7f07a41370a0 con 0x7f07a41430c0 2024-03-25T18:18:09.647 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.646+0000 7f07a0ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:18:09.647 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.646+0000 7f07a0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:09.647 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.646+0000 7f07a0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:09.647 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.646+0000 7f07a0ff9640 10 monclient(hunting): my global_id is 4305 2024-03-25T18:18:09.647 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.646+0000 7f07a0ff9640 1 -- 172.21.15.97:0/1087733458 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0790004eb0 con 0x7f07a41430c0 2024-03-25T18:18:09.649 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.648+0000 7f07a0ff9640 1 -- 172.21.15.97:0/1087733458 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1872417410 0 0) 0x7f07a4135220 con 0x7f07a40a98f0 2024-03-25T18:18:09.649 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.648+0000 7f07a0ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:18:09.649 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.648+0000 7f07a0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:09.649 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.648+0000 7f07a0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:09.649 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.648+0000 7f07a0ff9640 10 monclient(hunting): my global_id is 4295 2024-03-25T18:18:09.649 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.648+0000 7f07a0ff9640 1 -- 172.21.15.97:0/1087733458 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07a41370a0 con 0x7f07a40a98f0 2024-03-25T18:18:09.652 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.650+0000 7f07a0ff9640 1 -- 172.21.15.97:0/1087733458 <== mon.1 v1:172.21.15.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2519207175 0 0) 0x7f07a4135090 con 0x7f07a413f710 2024-03-25T18:18:09.652 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.650+0000 7f07a0ff9640 1 -- 172.21.15.97:0/1087733458 >> v1:172.21.15.97:6790/0 conn(0x7f07a40a98f0 legacy=0x7f07a41dae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:09.652 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.650+0000 7f07a0ff9640 1 -- 172.21.15.97:0/1087733458 >> v1:172.21.15.97:6789/0 conn(0x7f07a41430c0 legacy=0x7f07a41d48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:09.652 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.650+0000 7f07a0ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:18:09.652 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.650+0000 7f07a0ff9640 1 monclient: found mon.b 2024-03-25T18:18:09.652 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.650+0000 7f07a0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:09.652 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.650+0000 7f07a0ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:18:09.652 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.650+0000 7f07a0ff9640 1 -- 172.21.15.97:0/1087733458 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07a41d5000 con 0x7f07a413f710 2024-03-25T18:18:09.652 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.650+0000 7f07a0ff9640 10 monclient: _finish_auth 0 2024-03-25T18:18:09.652 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.650+0000 7f07a0ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:18:09.653 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.650+0000 7f07a0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:09.653 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.650+0000 7f07aa3e6640 5 monclient: authenticate success, global_id 4300 2024-03-25T18:18:09.653 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.650+0000 7f07aa3e6640 10 monclient: _renew_subs 2024-03-25T18:18:09.653 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.650+0000 7f07aa3e6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:18:09.653 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.650+0000 7f07aa3e6640 1 -- 172.21.15.97:0/1087733458 --> v1:172.21.15.98:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07a41d52b0 con 0x7f07a413f710 2024-03-25T18:18:09.653 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.650+0000 7f07aa3e6640 10 client.4300.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:18:09.653 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.650+0000 7f07aa3e6640 10 monclient: _renew_subs 2024-03-25T18:18:09.653 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.650+0000 7f07aa3e6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:18:09.653 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.650+0000 7f07aa3e6640 1 -- 172.21.15.97:0/1087733458 --> v1:172.21.15.98:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f07a4135910 con 0x7f07a413f710 2024-03-25T18:18:09.653 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.650+0000 7f07aa3e6640 1 librados: init done 2024-03-25T18:18:09.654 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.651+0000 7f07aa3e6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:09.654 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.651+0000 7f07aa3e6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:09.654 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.651+0000 7f07aa3e6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:18:09.654 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.651+0000 7f07aa3e6640 1 -- 172.21.15.97:0/1087733458 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07a40627b0 con 0x7f07a413f710 2024-03-25T18:18:09.918 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.916+0000 7f07a0ff9640 1 -- 172.21.15.97:0/1087733458 <== mon.1 v1:172.21.15.98:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0798004a90 con 0x7f07a413f710 2024-03-25T18:18:09.918 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.916+0000 7f07a0ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:09.918 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.916+0000 7f07a0ff9640 1 -- 172.21.15.97:0/1087733458 <== mon.1 v1:172.21.15.98:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f0798004e60 con 0x7f07a413f710 2024-03-25T18:18:09.918 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.916+0000 7f07a0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:09.918 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.916+0000 7f07a0ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:18:09.918 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.916+0000 7f07a0ff9640 10 monclient: dump: 2024-03-25T18:18:09.918 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:18:09.918 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:18:09.918 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:18:09.919 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:18:09.919 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:09.919 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:18:09.919 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:18:09.919 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:18:09.919 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:18:09.919 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:18:09.919 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.916+0000 7f07a27fc640 4 set_mon_vals no callback set 2024-03-25T18:18:09.920 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.917+0000 7f07a0ff9640 1 -- 172.21.15.97:0/1087733458 <== mon.1 v1:172.21.15.98:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2377622137 0 0) 0x7f0798005080 con 0x7f07a413f710 2024-03-25T18:18:09.920 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.918+0000 7f07a3fff640 10 monclient: get_auth_request con 0x7f079006de80 auth_method 0 2024-03-25T18:18:09.920 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.918+0000 7f07a0ff9640 10 client.4300.objecter ms_handle_connect 0x7f079006de80 2024-03-25T18:18:09.923 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.922+0000 7f07a0ff9640 1 -- 172.21.15.97:0/1087733458 <== mon.1 v1:172.21.15.98:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 430711918 0 0) 0x7f07980549b0 con 0x7f07a413f710 2024-03-25T18:18:09.924 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.922+0000 7f07a0ff9640 10 client.4300.objecter ms_dispatch 0x7f07a4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:18:09.924 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.922+0000 7f07a0ff9640 3 client.4300.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:18:09.924 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.922+0000 7f07a0ff9640 3 client.4300.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:18:09.924 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.922+0000 7f07a0ff9640 20 client.4300.objecter dump_active .. 0 homeless 2024-03-25T18:18:09.924 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.922+0000 7f07a0ff9640 1 -- 172.21.15.97:0/1087733458 <== mon.1 v1:172.21.15.98:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f07980504c0 con 0x7f07a413f710 2024-03-25T18:18:09.924 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.922+0000 7f07a0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:09.924 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:09.922+0000 7f07a0ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:18:10.175 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:10.174+0000 7f07aa3e6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:10.175 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:10.174+0000 7f07aa3e6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:10.175 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:10.174+0000 7f07aa3e6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:18:10.176 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:10.174+0000 7f07aa3e6640 1 -- 172.21.15.97:0/1087733458 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f07a413bfe0 con 0x7f07a413f710 2024-03-25T18:18:10.195 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:10.194+0000 7f07a0ff9640 1 -- 172.21.15.97:0/1087733458 <== mon.1 v1:172.21.15.98:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2169427412) 0x7f07980523f0 con 0x7f07a413f710 2024-03-25T18:18:10.195 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:10.194+0000 7f07a0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:10.195 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:10.194+0000 7f07a0ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:18:10.196 INFO:teuthology.misc.health.smithi097.stdout: 2024-03-25T18:18:10.196 INFO:teuthology.misc.health.smithi097.stdout:{"epoch":10,"fsid":"4c3d472c-79b8-486b-9e3d-2738d4306754","created":"2024-03-25T18:16:14.737309+0000","modified":"2024-03-25T18:16:20.357492+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:20.357492+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cb5152b9-262f-45c3-94b8-18d0e81df558","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6daf8793-0838-473d-a7d6-9a0d2894c75d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"136f1493-3b9c-4f43-a2e7-641482d69dae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"397d538c-4f64-4daf-9e8b-662d09f91327","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4f1df08-663f-4bda-93f8-3878ab7c24b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c4b18d70-a3ce-4b15-9b70-eae5525bd761","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8dd4033a-de0e-4f8a-96dc-59f3c1f825b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0077722e-ffa6-4a3d-9891-929d9869b286","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:18:10.198 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:10.196+0000 7f07aa3e6640 10 librados: watch_flush enter 2024-03-25T18:18:10.198 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:10.196+0000 7f07aa3e6640 10 librados: watch_flush exit 2024-03-25T18:18:10.199 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:10.196+0000 7f07aa3e6640 20 client.4300.objecter shutdown clearing up homeless session... 2024-03-25T18:18:10.199 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:10.196+0000 7f07aa3e6640 10 client.4300.objecter successfully canceled tick 2024-03-25T18:18:10.199 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:10.197+0000 7f07aa3e6640 1 -- 172.21.15.97:0/1087733458 >> v1:172.21.15.98:6800/800387573 conn(0x7f079006de80 legacy=0x7f0790070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:10.199 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:10.197+0000 7f07aa3e6640 10 monclient: shutdown 2024-03-25T18:18:10.199 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:10.197+0000 7f07aa3e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:10.199 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:10.197+0000 7f07aa3e6640 1 -- 172.21.15.97:0/1087733458 >> v1:172.21.15.98:6789/0 conn(0x7f07a413f710 legacy=0x7f07a41d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:10.199 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:10.197+0000 7f07aa3e6640 1 -- 172.21.15.97:0/1087733458 shutdown_connections 2024-03-25T18:18:10.199 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:10.197+0000 7f07aa3e6640 1 -- 172.21.15.97:0/1087733458 >> 172.21.15.97:0/1087733458 conn(0x7f07a40b7ea0 msgr2=0x7f07a4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:10.199 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:10.197+0000 7f07aa3e6640 1 -- 172.21.15.97:0/1087733458 shutdown_connections 2024-03-25T18:18:10.199 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:10.197+0000 7f07aa3e6640 1 -- 172.21.15.97:0/1087733458 wait complete. 2024-03-25T18:18:10.199 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:10.197+0000 7f07aa3e6640 1 librados: shutdown 2024-03-25T18:18:10.210 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:18:12.065 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:02:14.104 34931== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:12.076 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:02:14.115 34931== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:12.116 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:02:14.155 34931== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:12.127 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:02:14.170 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:12.131 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:02:14.174 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:12.152 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:02:14.191 34935== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:12.156 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:02:14.195 34935== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:12.158 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:02:14.202 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:12.192 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:02:14.231 34935== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:17.017 DEBUG:teuthology.orchestra.run.smithi097:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:18:17.134 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.133+0000 7fae289cf640 10 monclient: get_monmap_and_config 2024-03-25T18:18:17.135 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.133+0000 7fae289cf640 10 monclient: build_initial_monmap 2024-03-25T18:18:17.135 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.133+0000 7fae289cf640 10 monclient: monmap: 2024-03-25T18:18:17.135 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:18:17.135 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:18:17.135 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:18:17.134353+0000 2024-03-25T18:18:17.135 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:18:17.134353+0000 2024-03-25T18:18:17.135 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:17.135 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:18:17.135 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:18:17.135 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T18:18:17.135 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-b 2024-03-25T18:18:17.136 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:18:17.136 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.133+0000 7fae289cf640 1 Processor -- start 2024-03-25T18:18:17.136 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.133+0000 7fae289cf640 1 -- start start 2024-03-25T18:18:17.136 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.133+0000 7fae289cf640 10 monclient: init 2024-03-25T18:18:17.137 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.134+0000 7fae289cf640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:17.137 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.134+0000 7fae289cf640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:18:17.137 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.134+0000 7fae289cf640 10 monclient(hunting): picked mon.noname-c con 0x7fae240a98f0 addr v1:172.21.15.98:6789/0 2024-03-25T18:18:17.137 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.134+0000 7fae289cf640 10 monclient(hunting): picked mon.noname-a con 0x7fae2413f710 addr v1:172.21.15.97:6789/0 2024-03-25T18:18:17.137 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.134+0000 7fae289cf640 10 monclient(hunting): picked mon.noname-b con 0x7fae241430c0 addr v1:172.21.15.97:6790/0 2024-03-25T18:18:17.137 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.134+0000 7fae289cf640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae24142520 con 0x7fae2413f710 2024-03-25T18:18:17.137 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.134+0000 7fae289cf640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae240b28a0 con 0x7fae240a98f0 2024-03-25T18:18:17.137 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.135+0000 7fae289cf640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae240b2a30 con 0x7fae241430c0 2024-03-25T18:18:17.137 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.135+0000 7fae289cf640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:17.137 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.135+0000 7fae289cf640 10 monclient(hunting): authenticate will time out at 7709.410896s 2024-03-25T18:18:17.146 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.145+0000 7fae22d9d640 1 --1- >> v1:172.21.15.97:6790/0 conn(0x7fae241430c0 0x7fae2414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6790/0 says I am v1:172.21.15.97:53484/0 (socket says 172.21.15.97:53484) 2024-03-25T18:18:17.146 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.145+0000 7fae22d9d640 1 -- 172.21.15.97:0/1844558724 learned_addr learned my addr 172.21.15.97:0/1844558724 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:18:17.157 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.156+0000 7fae2159a640 1 -- 172.21.15.97:0/1844558724 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2684881010 0 0) 0x7fae240b28a0 con 0x7fae240a98f0 2024-03-25T18:18:17.157 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.156+0000 7fae2159a640 10 monclient(hunting): _negotiate 2024-03-25T18:18:17.157 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.156+0000 7fae2159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:17.157 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.156+0000 7fae2159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:17.157 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.156+0000 7fae2159a640 10 monclient(hunting): my global_id is 4303 2024-03-25T18:18:17.157 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.156+0000 7fae2159a640 1 -- 172.21.15.97:0/1844558724 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae10003520 con 0x7fae240a98f0 2024-03-25T18:18:17.159 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.157+0000 7fae2159a640 1 -- 172.21.15.97:0/1844558724 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3289471430 0 0) 0x7fae240b2a30 con 0x7fae241430c0 2024-03-25T18:18:17.159 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.157+0000 7fae2159a640 10 monclient(hunting): _negotiate 2024-03-25T18:18:17.159 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.157+0000 7fae2159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:17.159 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.157+0000 7fae2159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:17.159 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.157+0000 7fae2159a640 10 monclient(hunting): my global_id is 4298 2024-03-25T18:18:17.159 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.157+0000 7fae2159a640 1 -- 172.21.15.97:0/1844558724 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae240b28a0 con 0x7fae241430c0 2024-03-25T18:18:17.161 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.159+0000 7fae2159a640 1 -- 172.21.15.97:0/1844558724 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3093271329 0 0) 0x7fae24142520 con 0x7fae2413f710 2024-03-25T18:18:17.161 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.159+0000 7fae2159a640 10 monclient(hunting): _negotiate 2024-03-25T18:18:17.161 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.159+0000 7fae2159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:17.161 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.159+0000 7fae2159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:17.161 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.159+0000 7fae2159a640 10 monclient(hunting): my global_id is 4308 2024-03-25T18:18:17.161 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.159+0000 7fae2159a640 1 -- 172.21.15.97:0/1844558724 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae240b2a30 con 0x7fae2413f710 2024-03-25T18:18:17.168 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.167+0000 7fae2159a640 1 -- 172.21.15.97:0/1844558724 <== mon.2 v1:172.21.15.97:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3892384815 0 0) 0x7fae240b28a0 con 0x7fae241430c0 2024-03-25T18:18:17.168 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.167+0000 7fae2159a640 1 -- 172.21.15.97:0/1844558724 >> v1:172.21.15.98:6789/0 conn(0x7fae240a98f0 legacy=0x7fae240a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:17.168 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.167+0000 7fae2159a640 1 -- 172.21.15.97:0/1844558724 >> v1:172.21.15.97:6789/0 conn(0x7fae2413f710 legacy=0x7fae241429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:17.168 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.167+0000 7fae2159a640 10 monclient: _finish_hunting 0 2024-03-25T18:18:17.169 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.167+0000 7fae2159a640 1 monclient: found mon.noname-b 2024-03-25T18:18:17.169 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.167+0000 7fae2159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:17.169 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.167+0000 7fae2159a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.97:6790/0 2024-03-25T18:18:17.169 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.167+0000 7fae2159a640 1 -- 172.21.15.97:0/1844558724 --> v1:172.21.15.97:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae240b44c0 con 0x7fae241430c0 2024-03-25T18:18:17.169 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.167+0000 7fae2159a640 10 monclient: _finish_auth 0 2024-03-25T18:18:17.169 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.167+0000 7fae2159a640 10 monclient: _check_auth_tickets 2024-03-25T18:18:17.169 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.167+0000 7fae2159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:17.169 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.167+0000 7fae2159a640 1 -- 172.21.15.97:0/1844558724 <== mon.2 v1:172.21.15.97:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7fae0c001670 con 0x7fae241430c0 2024-03-25T18:18:17.169 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.167+0000 7fae289cf640 5 monclient: authenticate success, global_id 4298 2024-03-25T18:18:17.169 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.167+0000 7fae289cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:18:17.169 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.167+0000 7fae2159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:17.169 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.167+0000 7fae2159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:18:17.169 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.167+0000 7fae2159a640 10 monclient: dump: 2024-03-25T18:18:17.170 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:18:17.170 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:18:17.170 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:18:17.170 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:18:17.170 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:17.170 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:18:17.170 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:18:17.170 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:18:17.170 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:18:17.170 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:18:17.170 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.167+0000 7fae289cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:18:17.179 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.177+0000 7fae2159a640 1 -- 172.21.15.97:0/1844558724 <== mon.2 v1:172.21.15.97:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fae0c0017d0 con 0x7fae241430c0 2024-03-25T18:18:17.179 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.177+0000 7fae2159a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:17.179 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.177+0000 7fae2159a640 1 -- 172.21.15.97:0/1844558724 <== mon.2 v1:172.21.15.97:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7fae0c001be0 con 0x7fae241430c0 2024-03-25T18:18:17.179 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.177+0000 7fae289cf640 10 monclient: get_monmap_and_config success 2024-03-25T18:18:17.179 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.177+0000 7fae289cf640 4 set_mon_vals no callback set 2024-03-25T18:18:17.179 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.177+0000 7fae2159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:17.179 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.177+0000 7fae2159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:18:17.179 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.177+0000 7fae2159a640 10 monclient: dump: 2024-03-25T18:18:17.179 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:18:17.179 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:18:17.179 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:18:17.179 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:18:17.180 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:17.180 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:18:17.180 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:18:17.180 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:18:17.180 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:18:17.180 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:18:17.182 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.178+0000 7fae289cf640 10 monclient: shutdown 2024-03-25T18:18:17.182 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.178+0000 7fae289cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:17.182 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.178+0000 7fae289cf640 1 -- 172.21.15.97:0/1844558724 >> v1:172.21.15.97:6790/0 conn(0x7fae241430c0 legacy=0x7fae2414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:17.182 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.178+0000 7fae289cf640 1 -- 172.21.15.97:0/1844558724 shutdown_connections 2024-03-25T18:18:17.182 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.178+0000 7fae289cf640 1 -- 172.21.15.97:0/1844558724 >> 172.21.15.97:0/1844558724 conn(0x7fae240b7ea0 msgr2=0x7fae240b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:17.182 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.178+0000 7fae289cf640 1 -- 172.21.15.97:0/1844558724 shutdown_connections 2024-03-25T18:18:17.183 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.178+0000 7fae289cf640 1 -- 172.21.15.97:0/1844558724 wait complete. 2024-03-25T18:18:17.183 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.179+0000 7fae289cf640 10 monclient: build_initial_monmap 2024-03-25T18:18:17.183 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.179+0000 7fae289cf640 10 monclient: monmap: 2024-03-25T18:18:17.183 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:18:17.183 INFO:teuthology.misc.health.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:18:17.183 INFO:teuthology.misc.health.smithi097.stderr:last_changed 0.000000 2024-03-25T18:18:17.183 INFO:teuthology.misc.health.smithi097.stderr:created 0.000000 2024-03-25T18:18:17.183 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:17.183 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 1 2024-03-25T18:18:17.183 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:18:17.183 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T18:18:17.183 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-c 2024-03-25T18:18:17.183 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:18:17.184 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.179+0000 7fae289cf640 1 librados: starting msgr at 2024-03-25T18:18:17.184 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.179+0000 7fae289cf640 1 librados: starting objecter 2024-03-25T18:18:17.184 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.179+0000 7fae289cf640 1 Processor -- start 2024-03-25T18:18:17.184 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.180+0000 7fae289cf640 1 -- start start 2024-03-25T18:18:17.184 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.180+0000 7fae289cf640 1 librados: setting wanted keys 2024-03-25T18:18:17.184 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.180+0000 7fae289cf640 1 librados: calling monclient init 2024-03-25T18:18:17.184 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.180+0000 7fae289cf640 10 monclient: init 2024-03-25T18:18:17.184 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.180+0000 7fae289cf640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:17.184 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.180+0000 7fae289cf640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:18:17.184 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.180+0000 7fae289cf640 10 monclient(hunting): picked mon.noname-c con 0x7fae240a98f0 addr v1:172.21.15.97:6790/0 2024-03-25T18:18:17.184 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.180+0000 7fae289cf640 10 monclient(hunting): picked mon.noname-a con 0x7fae2413f710 addr v1:172.21.15.97:6789/0 2024-03-25T18:18:17.184 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.180+0000 7fae289cf640 10 monclient(hunting): picked mon.noname-b con 0x7fae241430c0 addr v1:172.21.15.98:6789/0 2024-03-25T18:18:17.184 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.180+0000 7fae289cf640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae24137440 con 0x7fae2413f710 2024-03-25T18:18:17.184 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.180+0000 7fae289cf640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae24137690 con 0x7fae241430c0 2024-03-25T18:18:17.185 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.180+0000 7fae289cf640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae241d7ca0 con 0x7fae240a98f0 2024-03-25T18:18:17.185 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.180+0000 7fae289cf640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:17.185 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.180+0000 7fae289cf640 10 monclient(hunting): authenticate will time out at 7709.456516s 2024-03-25T18:18:17.192 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.190+0000 7fae2259c640 1 --1- >> v1:172.21.15.97:6790/0 conn(0x7fae240a98f0 0x7fae24139400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.97:6790/0 says I am v1:172.21.15.97:53520/0 (socket says 172.21.15.97:53520) 2024-03-25T18:18:17.192 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.190+0000 7fae2259c640 1 -- 172.21.15.97:0/3530507851 learned_addr learned my addr 172.21.15.97:0/3530507851 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:18:17.193 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.192+0000 7fae1affd640 10 client.?.objecter ms_handle_connect 0x7fae240a98f0 2024-03-25T18:18:17.193 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.192+0000 7fae1affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:17.198 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.196+0000 7fae1affd640 10 client.?.objecter ms_handle_connect 0x7fae2413f710 2024-03-25T18:18:17.198 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.197+0000 7fae1affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:17.208 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.207+0000 7fae1affd640 10 client.?.objecter ms_handle_connect 0x7fae241430c0 2024-03-25T18:18:17.208 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.207+0000 7fae1affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:17.210 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.208+0000 7fae1affd640 1 -- 172.21.15.97:0/3530507851 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1714600746 0 0) 0x7fae24137440 con 0x7fae2413f710 2024-03-25T18:18:17.210 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.208+0000 7fae1affd640 10 monclient(hunting): _negotiate 2024-03-25T18:18:17.210 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.208+0000 7fae1affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:17.210 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.208+0000 7fae1affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:17.210 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.208+0000 7fae1affd640 10 monclient(hunting): my global_id is 4311 2024-03-25T18:18:17.210 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.208+0000 7fae1affd640 1 -- 172.21.15.97:0/3530507851 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae04004eb0 con 0x7fae2413f710 2024-03-25T18:18:17.212 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.211+0000 7fae1affd640 1 -- 172.21.15.97:0/3530507851 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2278181814 0 0) 0x7fae241d7ca0 con 0x7fae240a98f0 2024-03-25T18:18:17.212 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.211+0000 7fae1affd640 10 monclient(hunting): _negotiate 2024-03-25T18:18:17.213 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.211+0000 7fae1affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:17.213 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.211+0000 7fae1affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:17.213 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.211+0000 7fae1affd640 10 monclient(hunting): my global_id is 4301 2024-03-25T18:18:17.213 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.211+0000 7fae1affd640 1 -- 172.21.15.97:0/3530507851 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae24137440 con 0x7fae240a98f0 2024-03-25T18:18:17.215 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.214+0000 7fae1affd640 1 -- 172.21.15.97:0/3530507851 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 384994114 0 0) 0x7fae24137690 con 0x7fae241430c0 2024-03-25T18:18:17.215 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.214+0000 7fae1affd640 10 monclient(hunting): _negotiate 2024-03-25T18:18:17.215 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.214+0000 7fae1affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:17.215 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.214+0000 7fae1affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:17.215 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.214+0000 7fae1affd640 10 monclient(hunting): my global_id is 4306 2024-03-25T18:18:17.215 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.214+0000 7fae1affd640 1 -- 172.21.15.97:0/3530507851 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae241d7ca0 con 0x7fae241430c0 2024-03-25T18:18:17.223 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.221+0000 7fae1affd640 1 -- 172.21.15.97:0/3530507851 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4125756788 0 0) 0x7fae04004eb0 con 0x7fae2413f710 2024-03-25T18:18:17.223 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.221+0000 7fae1affd640 1 -- 172.21.15.97:0/3530507851 --> v1:172.21.15.97:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fae24137690 con 0x7fae2413f710 2024-03-25T18:18:17.223 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.221+0000 7fae1affd640 1 -- 172.21.15.97:0/3530507851 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7fae140018f0 con 0x7fae2413f710 2024-03-25T18:18:17.223 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.221+0000 7fae1affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:17.223 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.221+0000 7fae1affd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:18:17.223 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.221+0000 7fae1affd640 10 monclient(hunting): dump: 2024-03-25T18:18:17.223 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:18:17.223 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:18:17.224 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:18:17.224 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:18:17.224 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:17.224 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:18:17.224 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:18:17.224 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:18:17.224 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:18:17.224 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:18:17.224 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.221+0000 7fae1affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:18:17.235 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.234+0000 7fae1affd640 1 -- 172.21.15.97:0/3530507851 <== mon.0 v1:172.21.15.97:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1086239947 0 0) 0x7fae24137690 con 0x7fae2413f710 2024-03-25T18:18:17.235 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.234+0000 7fae1affd640 1 -- 172.21.15.97:0/3530507851 >> v1:172.21.15.97:6790/0 conn(0x7fae240a98f0 legacy=0x7fae24139400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:17.235 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.234+0000 7fae1affd640 1 -- 172.21.15.97:0/3530507851 >> v1:172.21.15.98:6789/0 conn(0x7fae241430c0 legacy=0x7fae24132f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:17.236 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.234+0000 7fae1affd640 10 monclient: _finish_hunting 0 2024-03-25T18:18:17.236 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.234+0000 7fae1affd640 1 monclient: found mon.a 2024-03-25T18:18:17.236 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.234+0000 7fae1affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:17.236 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.234+0000 7fae1affd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:18:17.236 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.234+0000 7fae1affd640 1 -- 172.21.15.97:0/3530507851 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae24136b60 con 0x7fae2413f710 2024-03-25T18:18:17.236 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.234+0000 7fae1affd640 10 monclient: _finish_auth 0 2024-03-25T18:18:17.236 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.234+0000 7fae1affd640 10 monclient: _check_auth_tickets 2024-03-25T18:18:17.236 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.234+0000 7fae1affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:17.236 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.234+0000 7fae289cf640 5 monclient: authenticate success, global_id 4311 2024-03-25T18:18:17.236 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.234+0000 7fae289cf640 10 monclient: _renew_subs 2024-03-25T18:18:17.236 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.234+0000 7fae289cf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:18:17.236 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.234+0000 7fae289cf640 1 -- 172.21.15.97:0/3530507851 --> v1:172.21.15.97:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae24137bb0 con 0x7fae2413f710 2024-03-25T18:18:17.236 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.234+0000 7fae289cf640 10 client.4311.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:18:17.237 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.234+0000 7fae289cf640 10 monclient: _renew_subs 2024-03-25T18:18:17.237 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.234+0000 7fae289cf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:18:17.237 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.234+0000 7fae289cf640 1 -- 172.21.15.97:0/3530507851 --> v1:172.21.15.97:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fae240b7330 con 0x7fae2413f710 2024-03-25T18:18:17.237 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.234+0000 7fae289cf640 1 librados: init done 2024-03-25T18:18:17.237 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.235+0000 7fae289cf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:17.237 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.235+0000 7fae289cf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:17.238 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.235+0000 7fae289cf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:18:17.238 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.235+0000 7fae289cf640 1 -- 172.21.15.97:0/3530507851 --> v1:172.21.15.97:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae24137f00 con 0x7fae2413f710 2024-03-25T18:18:17.545 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.543+0000 7fae1affd640 1 -- 172.21.15.97:0/3530507851 <== mon.0 v1:172.21.15.97:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fae14003610 con 0x7fae2413f710 2024-03-25T18:18:17.545 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.543+0000 7fae1affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:17.545 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.543+0000 7fae1affd640 1 -- 172.21.15.97:0/3530507851 <== mon.0 v1:172.21.15.97:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7fae140039a0 con 0x7fae2413f710 2024-03-25T18:18:17.545 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.543+0000 7fae1affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:17.545 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.543+0000 7fae1affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:18:17.545 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.543+0000 7fae1affd640 10 monclient: dump: 2024-03-25T18:18:17.545 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:18:17.545 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:18:17.545 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:18:17.545 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:18:17.545 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:17.546 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:18:17.546 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:18:17.546 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:18:17.546 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:18:17.546 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:18:17.546 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.543+0000 7fae20d99640 4 set_mon_vals no callback set 2024-03-25T18:18:17.546 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.544+0000 7fae1affd640 1 -- 172.21.15.97:0/3530507851 <== mon.0 v1:172.21.15.97:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2377622137 0 0) 0x7fae14003bc0 con 0x7fae2413f710 2024-03-25T18:18:17.546 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.545+0000 7fae2259c640 10 monclient: get_auth_request con 0x7fae0406de90 auth_method 0 2024-03-25T18:18:17.547 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.545+0000 7fae1affd640 10 client.4311.objecter ms_handle_connect 0x7fae0406de90 2024-03-25T18:18:17.552 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.551+0000 7fae1affd640 1 -- 172.21.15.97:0/3530507851 <== mon.0 v1:172.21.15.97:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 430711918 0 0) 0x7fae14053600 con 0x7fae2413f710 2024-03-25T18:18:17.552 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.551+0000 7fae1affd640 10 client.4311.objecter ms_dispatch 0x7fae24000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:18:17.552 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.551+0000 7fae1affd640 3 client.4311.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:18:17.552 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.551+0000 7fae1affd640 3 client.4311.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:18:17.552 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.551+0000 7fae1affd640 20 client.4311.objecter dump_active .. 0 homeless 2024-03-25T18:18:17.553 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.551+0000 7fae1affd640 1 -- 172.21.15.97:0/3530507851 <== mon.0 v1:172.21.15.97:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fae1404f110 con 0x7fae2413f710 2024-03-25T18:18:17.553 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.551+0000 7fae1affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:17.553 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.551+0000 7fae1affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:18:17.790 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.789+0000 7fae289cf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:17.790 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.789+0000 7fae289cf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:17.790 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.789+0000 7fae289cf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:18:17.790 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.789+0000 7fae289cf640 1 -- 172.21.15.97:0/3530507851 --> v1:172.21.15.97:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fae24136cf0 con 0x7fae2413f710 2024-03-25T18:18:17.821 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.820+0000 7fae1affd640 1 -- 172.21.15.97:0/3530507851 <== mon.0 v1:172.21.15.97:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2169427412) 0x7fae14051040 con 0x7fae2413f710 2024-03-25T18:18:17.821 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.820+0000 7fae1affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:17.821 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.820+0000 7fae1affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:18:17.821 INFO:teuthology.misc.health.smithi097.stdout: 2024-03-25T18:18:17.821 INFO:teuthology.misc.health.smithi097.stdout:{"epoch":10,"fsid":"4c3d472c-79b8-486b-9e3d-2738d4306754","created":"2024-03-25T18:16:14.737309+0000","modified":"2024-03-25T18:16:20.357492+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:20.357492+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cb5152b9-262f-45c3-94b8-18d0e81df558","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6daf8793-0838-473d-a7d6-9a0d2894c75d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"136f1493-3b9c-4f43-a2e7-641482d69dae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"397d538c-4f64-4daf-9e8b-662d09f91327","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4f1df08-663f-4bda-93f8-3878ab7c24b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c4b18d70-a3ce-4b15-9b70-eae5525bd761","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8dd4033a-de0e-4f8a-96dc-59f3c1f825b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0077722e-ffa6-4a3d-9891-929d9869b286","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:18:17.824 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.822+0000 7fae289cf640 10 librados: watch_flush enter 2024-03-25T18:18:17.824 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.822+0000 7fae289cf640 10 librados: watch_flush exit 2024-03-25T18:18:17.825 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.822+0000 7fae289cf640 20 client.4311.objecter shutdown clearing up homeless session... 2024-03-25T18:18:17.825 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.822+0000 7fae289cf640 10 client.4311.objecter successfully canceled tick 2024-03-25T18:18:17.825 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.822+0000 7fae289cf640 1 -- 172.21.15.97:0/3530507851 >> v1:172.21.15.98:6800/800387573 conn(0x7fae0406de90 legacy=0x7fae04070350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:17.825 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.822+0000 7fae289cf640 10 monclient: shutdown 2024-03-25T18:18:17.825 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.823+0000 7fae289cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:17.825 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.823+0000 7fae289cf640 1 -- 172.21.15.97:0/3530507851 >> v1:172.21.15.97:6789/0 conn(0x7fae2413f710 legacy=0x7fae241d7590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:17.825 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.823+0000 7fae289cf640 1 -- 172.21.15.97:0/3530507851 shutdown_connections 2024-03-25T18:18:17.825 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.823+0000 7fae289cf640 1 -- 172.21.15.97:0/3530507851 >> 172.21.15.97:0/3530507851 conn(0x7fae240b7ea0 msgr2=0x7fae24074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:17.825 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.823+0000 7fae289cf640 1 -- 172.21.15.97:0/3530507851 shutdown_connections 2024-03-25T18:18:17.825 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.823+0000 7fae289cf640 1 -- 172.21.15.97:0/3530507851 wait complete. 2024-03-25T18:18:17.825 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:17.823+0000 7fae289cf640 1 librados: shutdown 2024-03-25T18:18:17.834 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:18:24.640 DEBUG:teuthology.orchestra.run.smithi097:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:18:24.757 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.756+0000 7f8f9fb89640 10 monclient: get_monmap_and_config 2024-03-25T18:18:24.758 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.756+0000 7f8f9fb89640 10 monclient: build_initial_monmap 2024-03-25T18:18:24.758 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.756+0000 7f8f9fb89640 10 monclient: monmap: 2024-03-25T18:18:24.758 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:18:24.758 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:18:24.758 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:18:24.758056+0000 2024-03-25T18:18:24.758 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:18:24.758056+0000 2024-03-25T18:18:24.758 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:24.758 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:18:24.758 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:18:24.758 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T18:18:24.758 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-b 2024-03-25T18:18:24.758 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:18:24.759 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.757+0000 7f8f9fb89640 1 Processor -- start 2024-03-25T18:18:24.759 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.757+0000 7f8f9fb89640 1 -- start start 2024-03-25T18:18:24.759 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.757+0000 7f8f9fb89640 10 monclient: init 2024-03-25T18:18:24.759 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.758+0000 7f8f9fb89640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:24.760 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.758+0000 7f8f9fb89640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:18:24.760 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.758+0000 7f8f9fb89640 10 monclient(hunting): picked mon.noname-a con 0x7f8f980a98f0 addr v1:172.21.15.97:6789/0 2024-03-25T18:18:24.760 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.758+0000 7f8f9fb89640 10 monclient(hunting): picked mon.noname-b con 0x7f8f9813f710 addr v1:172.21.15.97:6790/0 2024-03-25T18:18:24.760 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.758+0000 7f8f9fb89640 10 monclient(hunting): picked mon.noname-c con 0x7f8f981430c0 addr v1:172.21.15.98:6789/0 2024-03-25T18:18:24.760 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.758+0000 7f8f9fb89640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f98142520 con 0x7f8f980a98f0 2024-03-25T18:18:24.760 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.758+0000 7f8f9fb89640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f980b28a0 con 0x7f8f981430c0 2024-03-25T18:18:24.760 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.758+0000 7f8f9fb89640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f980b2a30 con 0x7f8f9813f710 2024-03-25T18:18:24.760 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.758+0000 7f8f9fb89640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:24.760 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.758+0000 7f8f9fb89640 10 monclient(hunting): authenticate will time out at 7717.034642s 2024-03-25T18:18:24.785 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.784+0000 7f8f9d124640 1 --1- >> v1:172.21.15.97:6790/0 conn(0x7f8f9813f710 0x7f8f981429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6790/0 says I am v1:172.21.15.97:32990/0 (socket says 172.21.15.97:32990) 2024-03-25T18:18:24.785 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.784+0000 7f8f9d124640 1 -- 172.21.15.97:0/1846266633 learned_addr learned my addr 172.21.15.97:0/1846266633 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:18:24.792 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.790+0000 7f8f9c923640 1 -- 172.21.15.97:0/1846266633 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4119902552 0 0) 0x7f8f980b28a0 con 0x7f8f981430c0 2024-03-25T18:18:24.792 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.790+0000 7f8f9c923640 10 monclient(hunting): _negotiate 2024-03-25T18:18:24.792 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.790+0000 7f8f9c923640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:24.792 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.790+0000 7f8f9c923640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:24.792 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.790+0000 7f8f9c923640 10 monclient(hunting): my global_id is 4309 2024-03-25T18:18:24.792 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.790+0000 7f8f9c923640 1 -- 172.21.15.97:0/1846266633 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f80003520 con 0x7f8f981430c0 2024-03-25T18:18:24.800 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.799+0000 7f8f9c923640 1 -- 172.21.15.97:0/1846266633 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1834362868 0 0) 0x7f8f98142520 con 0x7f8f980a98f0 2024-03-25T18:18:24.800 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.799+0000 7f8f9c923640 10 monclient(hunting): _negotiate 2024-03-25T18:18:24.800 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.799+0000 7f8f9c923640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:24.800 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.799+0000 7f8f9c923640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:24.800 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.799+0000 7f8f9c923640 10 monclient(hunting): my global_id is 4314 2024-03-25T18:18:24.800 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.799+0000 7f8f9c923640 1 -- 172.21.15.97:0/1846266633 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f980b28a0 con 0x7f8f980a98f0 2024-03-25T18:18:24.803 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.802+0000 7f8f9c923640 1 -- 172.21.15.97:0/1846266633 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 794997428 0 0) 0x7f8f980b2a30 con 0x7f8f9813f710 2024-03-25T18:18:24.803 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.802+0000 7f8f9c923640 10 monclient(hunting): _negotiate 2024-03-25T18:18:24.803 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.802+0000 7f8f9c923640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:24.803 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.802+0000 7f8f9c923640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:24.803 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.802+0000 7f8f9c923640 10 monclient(hunting): my global_id is 4304 2024-03-25T18:18:24.803 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.802+0000 7f8f9c923640 1 -- 172.21.15.97:0/1846266633 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f98142520 con 0x7f8f9813f710 2024-03-25T18:18:24.804 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.803+0000 7f8f9c923640 1 -- 172.21.15.97:0/1846266633 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4179952041 0 0) 0x7f8f80003520 con 0x7f8f981430c0 2024-03-25T18:18:24.804 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.803+0000 7f8f9c923640 1 -- 172.21.15.97:0/1846266633 >> v1:172.21.15.97:6790/0 conn(0x7f8f9813f710 legacy=0x7f8f981429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:24.804 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.803+0000 7f8f9c923640 1 -- 172.21.15.97:0/1846266633 >> v1:172.21.15.97:6789/0 conn(0x7f8f980a98f0 legacy=0x7f8f980a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:24.804 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.803+0000 7f8f9c923640 10 monclient: _finish_hunting 0 2024-03-25T18:18:24.804 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.803+0000 7f8f9c923640 1 monclient: found mon.noname-c 2024-03-25T18:18:24.805 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.803+0000 7f8f9c923640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:24.805 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.803+0000 7f8f9c923640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.98:6789/0 2024-03-25T18:18:24.805 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.803+0000 7f8f9c923640 1 -- 172.21.15.97:0/1846266633 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f980b44c0 con 0x7f8f981430c0 2024-03-25T18:18:24.805 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.803+0000 7f8f9c923640 10 monclient: _finish_auth 0 2024-03-25T18:18:24.805 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.803+0000 7f8f9c923640 10 monclient: _check_auth_tickets 2024-03-25T18:18:24.805 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.803+0000 7f8f9c923640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:24.805 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.803+0000 7f8f9c923640 1 -- 172.21.15.97:0/1846266633 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f8f90001a10 con 0x7f8f981430c0 2024-03-25T18:18:24.805 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.803+0000 7f8f9c923640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:24.805 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.803+0000 7f8f9c923640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:18:24.805 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.803+0000 7f8f9c923640 10 monclient: dump: 2024-03-25T18:18:24.805 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:18:24.805 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:18:24.805 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:18:24.805 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:18:24.805 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:24.805 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:18:24.806 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:18:24.806 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:18:24.806 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:18:24.806 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:18:24.806 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.803+0000 7f8f9fb89640 5 monclient: authenticate success, global_id 4309 2024-03-25T18:18:24.806 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.803+0000 7f8f9fb89640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:18:24.814 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.813+0000 7f8f9c923640 1 -- 172.21.15.97:0/1846266633 <== mon.1 v1:172.21.15.98:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8f90001bf0 con 0x7f8f981430c0 2024-03-25T18:18:24.814 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.813+0000 7f8f9c923640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:24.814 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.813+0000 7f8f9c923640 1 -- 172.21.15.97:0/1846266633 <== mon.1 v1:172.21.15.98:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f8f90002000 con 0x7f8f981430c0 2024-03-25T18:18:24.814 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.813+0000 7f8f9c923640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:24.814 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.813+0000 7f8f9c923640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:18:24.815 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.813+0000 7f8f9c923640 10 monclient: dump: 2024-03-25T18:18:24.815 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:18:24.815 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:18:24.815 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:18:24.815 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:18:24.815 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:24.815 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:18:24.815 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:18:24.815 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:18:24.815 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:18:24.815 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:18:24.815 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.813+0000 7f8f9fb89640 10 monclient: get_monmap_and_config success 2024-03-25T18:18:24.815 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.813+0000 7f8f9fb89640 4 set_mon_vals no callback set 2024-03-25T18:18:24.817 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.814+0000 7f8f9fb89640 10 monclient: shutdown 2024-03-25T18:18:24.817 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.814+0000 7f8f9fb89640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:24.817 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.814+0000 7f8f9fb89640 1 -- 172.21.15.97:0/1846266633 >> v1:172.21.15.98:6789/0 conn(0x7f8f981430c0 legacy=0x7f8f9814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:24.817 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.814+0000 7f8f9fb89640 1 -- 172.21.15.97:0/1846266633 shutdown_connections 2024-03-25T18:18:24.817 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.814+0000 7f8f9fb89640 1 -- 172.21.15.97:0/1846266633 >> 172.21.15.97:0/1846266633 conn(0x7f8f980b7ea0 msgr2=0x7f8f980b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:24.817 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.814+0000 7f8f9fb89640 1 -- 172.21.15.97:0/1846266633 shutdown_connections 2024-03-25T18:18:24.817 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.814+0000 7f8f9fb89640 1 -- 172.21.15.97:0/1846266633 wait complete. 2024-03-25T18:18:24.817 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.815+0000 7f8f9fb89640 10 monclient: build_initial_monmap 2024-03-25T18:18:24.817 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.815+0000 7f8f9fb89640 10 monclient: monmap: 2024-03-25T18:18:24.817 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:18:24.817 INFO:teuthology.misc.health.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:18:24.817 INFO:teuthology.misc.health.smithi097.stderr:last_changed 0.000000 2024-03-25T18:18:24.817 INFO:teuthology.misc.health.smithi097.stderr:created 0.000000 2024-03-25T18:18:24.817 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:24.818 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 1 2024-03-25T18:18:24.818 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:18:24.818 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T18:18:24.818 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-c 2024-03-25T18:18:24.818 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:18:24.818 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.815+0000 7f8f9fb89640 1 librados: starting msgr at 2024-03-25T18:18:24.818 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.815+0000 7f8f9fb89640 1 librados: starting objecter 2024-03-25T18:18:24.818 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.815+0000 7f8f9fb89640 1 Processor -- start 2024-03-25T18:18:24.818 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.815+0000 7f8f9fb89640 1 -- start start 2024-03-25T18:18:24.818 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.815+0000 7f8f9fb89640 1 librados: setting wanted keys 2024-03-25T18:18:24.818 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.815+0000 7f8f9fb89640 1 librados: calling monclient init 2024-03-25T18:18:24.818 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.815+0000 7f8f9fb89640 10 monclient: init 2024-03-25T18:18:24.819 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.816+0000 7f8f9fb89640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:24.819 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.816+0000 7f8f9fb89640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:18:24.819 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.816+0000 7f8f9fb89640 10 monclient(hunting): picked mon.noname-a con 0x7f8f980a98f0 addr v1:172.21.15.97:6789/0 2024-03-25T18:18:24.819 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.816+0000 7f8f9fb89640 10 monclient(hunting): picked mon.noname-c con 0x7f8f9813f710 addr v1:172.21.15.97:6790/0 2024-03-25T18:18:24.819 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.816+0000 7f8f9fb89640 10 monclient(hunting): picked mon.noname-b con 0x7f8f981430c0 addr v1:172.21.15.98:6789/0 2024-03-25T18:18:24.820 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.816+0000 7f8f9fb89640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f98138a40 con 0x7f8f980a98f0 2024-03-25T18:18:24.820 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.816+0000 7f8f9fb89640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f980ab9e0 con 0x7f8f981430c0 2024-03-25T18:18:24.820 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.816+0000 7f8f9fb89640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f98138d00 con 0x7f8f9813f710 2024-03-25T18:18:24.820 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.816+0000 7f8f9fb89640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:24.820 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.816+0000 7f8f9fb89640 10 monclient(hunting): authenticate will time out at 7717.092546s 2024-03-25T18:18:24.821 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.820+0000 7f8f9d124640 1 --1- >> v1:172.21.15.97:6790/0 conn(0x7f8f9813f710 0x7f8f980ac050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.97:6790/0 says I am v1:172.21.15.97:33032/0 (socket says 172.21.15.97:33032) 2024-03-25T18:18:24.821 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.820+0000 7f8f9d124640 1 -- 172.21.15.97:0/4276446526 learned_addr learned my addr 172.21.15.97:0/4276446526 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:18:24.826 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.825+0000 7f8f967fc640 10 client.?.objecter ms_handle_connect 0x7f8f981430c0 2024-03-25T18:18:24.826 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.825+0000 7f8f967fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:24.832 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.831+0000 7f8f967fc640 10 client.?.objecter ms_handle_connect 0x7f8f9813f710 2024-03-25T18:18:24.832 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.831+0000 7f8f967fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:24.835 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.833+0000 7f8f967fc640 1 -- 172.21.15.97:0/4276446526 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1823062962 0 0) 0x7f8f980ab9e0 con 0x7f8f981430c0 2024-03-25T18:18:24.835 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.833+0000 7f8f967fc640 10 monclient(hunting): _negotiate 2024-03-25T18:18:24.835 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.833+0000 7f8f967fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:24.835 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.833+0000 7f8f967fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:24.835 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.833+0000 7f8f967fc640 10 monclient(hunting): my global_id is 4312 2024-03-25T18:18:24.835 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.834+0000 7f8f967fc640 1 -- 172.21.15.97:0/4276446526 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f84004eb0 con 0x7f8f981430c0 2024-03-25T18:18:24.837 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.836+0000 7f8f967fc640 10 client.?.objecter ms_handle_connect 0x7f8f980a98f0 2024-03-25T18:18:24.837 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.836+0000 7f8f967fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:24.839 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.838+0000 7f8f967fc640 1 -- 172.21.15.97:0/4276446526 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 926853499 0 0) 0x7f8f98138d00 con 0x7f8f9813f710 2024-03-25T18:18:24.840 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.838+0000 7f8f967fc640 10 monclient(hunting): _negotiate 2024-03-25T18:18:24.840 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.838+0000 7f8f967fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:24.840 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.838+0000 7f8f967fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:24.840 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.838+0000 7f8f967fc640 10 monclient(hunting): my global_id is 4307 2024-03-25T18:18:24.840 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.838+0000 7f8f967fc640 1 -- 172.21.15.97:0/4276446526 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f980ab9e0 con 0x7f8f9813f710 2024-03-25T18:18:24.846 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.845+0000 7f8f967fc640 1 -- 172.21.15.97:0/4276446526 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1239647309 0 0) 0x7f8f84004eb0 con 0x7f8f981430c0 2024-03-25T18:18:24.846 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.845+0000 7f8f967fc640 1 -- 172.21.15.97:0/4276446526 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8f98138d00 con 0x7f8f981430c0 2024-03-25T18:18:24.846 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.845+0000 7f8f967fc640 1 -- 172.21.15.97:0/4276446526 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f8f90001f00 con 0x7f8f981430c0 2024-03-25T18:18:24.847 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.845+0000 7f8f967fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:24.847 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.845+0000 7f8f967fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:18:24.847 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.845+0000 7f8f967fc640 10 monclient(hunting): dump: 2024-03-25T18:18:24.847 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:18:24.847 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:18:24.847 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:18:24.847 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:18:24.847 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:24.847 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:18:24.847 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:18:24.847 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:18:24.847 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:18:24.847 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:18:24.847 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.845+0000 7f8f967fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:18:24.849 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.848+0000 7f8f967fc640 1 -- 172.21.15.97:0/4276446526 <== mon.2 v1:172.21.15.97:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 446504526 0 0) 0x7f8f980ab9e0 con 0x7f8f9813f710 2024-03-25T18:18:24.849 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.848+0000 7f8f967fc640 1 -- 172.21.15.97:0/4276446526 --> v1:172.21.15.97:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8f84004eb0 con 0x7f8f9813f710 2024-03-25T18:18:24.849 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.848+0000 7f8f967fc640 1 -- 172.21.15.97:0/4276446526 <== mon.2 v1:172.21.15.97:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f8f8c0018f0 con 0x7f8f9813f710 2024-03-25T18:18:24.849 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.848+0000 7f8f967fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:24.849 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.848+0000 7f8f967fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:18:24.849 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.848+0000 7f8f967fc640 10 monclient(hunting): dump: 2024-03-25T18:18:24.849 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:18:24.849 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:18:24.849 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:18:24.850 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:18:24.850 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:24.850 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:18:24.850 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:18:24.850 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:18:24.850 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:18:24.850 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:18:24.852 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.851+0000 7f8f967fc640 1 -- 172.21.15.97:0/4276446526 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3166520389 0 0) 0x7f8f98138a40 con 0x7f8f980a98f0 2024-03-25T18:18:24.852 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.851+0000 7f8f967fc640 10 monclient(hunting): _negotiate 2024-03-25T18:18:24.852 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.851+0000 7f8f967fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:24.852 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.851+0000 7f8f967fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:24.852 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.851+0000 7f8f967fc640 10 monclient(hunting): my global_id is 4317 2024-03-25T18:18:24.852 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.851+0000 7f8f967fc640 1 -- 172.21.15.97:0/4276446526 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f980ab9e0 con 0x7f8f980a98f0 2024-03-25T18:18:24.857 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.855+0000 7f8f967fc640 1 -- 172.21.15.97:0/4276446526 <== mon.2 v1:172.21.15.97:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3462346654 0 0) 0x7f8f84004eb0 con 0x7f8f9813f710 2024-03-25T18:18:24.857 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.855+0000 7f8f967fc640 1 -- 172.21.15.97:0/4276446526 >> v1:172.21.15.98:6789/0 conn(0x7f8f981430c0 legacy=0x7f8f98138330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:24.857 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.855+0000 7f8f967fc640 1 -- 172.21.15.97:0/4276446526 >> v1:172.21.15.97:6789/0 conn(0x7f8f980a98f0 legacy=0x7f8f98136240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:24.857 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.855+0000 7f8f967fc640 10 monclient: _finish_hunting 0 2024-03-25T18:18:24.857 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.855+0000 7f8f967fc640 1 monclient: found mon.c 2024-03-25T18:18:24.857 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.855+0000 7f8f967fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:24.857 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.855+0000 7f8f967fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.97:6790/0 2024-03-25T18:18:24.857 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.855+0000 7f8f967fc640 1 -- 172.21.15.97:0/4276446526 --> v1:172.21.15.97:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f980abc30 con 0x7f8f9813f710 2024-03-25T18:18:24.858 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.855+0000 7f8f967fc640 10 monclient: _finish_auth 0 2024-03-25T18:18:24.858 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.855+0000 7f8f967fc640 10 monclient: _check_auth_tickets 2024-03-25T18:18:24.858 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.855+0000 7f8f967fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:24.858 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.855+0000 7f8f9fb89640 5 monclient: authenticate success, global_id 4307 2024-03-25T18:18:24.858 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.855+0000 7f8f9fb89640 10 monclient: _renew_subs 2024-03-25T18:18:24.858 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.855+0000 7f8f9fb89640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.97:6790/0 2024-03-25T18:18:24.858 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.855+0000 7f8f9fb89640 1 -- 172.21.15.97:0/4276446526 --> v1:172.21.15.97:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f98139980 con 0x7f8f9813f710 2024-03-25T18:18:24.858 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.856+0000 7f8f9fb89640 10 client.4307.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:18:24.858 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.856+0000 7f8f9fb89640 10 monclient: _renew_subs 2024-03-25T18:18:24.858 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.856+0000 7f8f9fb89640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.97:6790/0 2024-03-25T18:18:24.858 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.856+0000 7f8f9fb89640 1 -- 172.21.15.97:0/4276446526 --> v1:172.21.15.97:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f981e32a0 con 0x7f8f9813f710 2024-03-25T18:18:24.858 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.856+0000 7f8f9fb89640 1 librados: init done 2024-03-25T18:18:24.859 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.857+0000 7f8f9fb89640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:24.859 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.857+0000 7f8f9fb89640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:24.859 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.857+0000 7f8f9fb89640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.97:6790/0 2024-03-25T18:18:24.859 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:24.857+0000 7f8f9fb89640 1 -- 172.21.15.97:0/4276446526 --> v1:172.21.15.97:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f980627b0 con 0x7f8f9813f710 2024-03-25T18:18:25.176 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:25.175+0000 7f8f967fc640 1 -- 172.21.15.97:0/4276446526 <== mon.2 v1:172.21.15.97:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8f8c0047e0 con 0x7f8f9813f710 2024-03-25T18:18:25.176 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:25.175+0000 7f8f967fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:25.176 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:25.175+0000 7f8f967fc640 1 -- 172.21.15.97:0/4276446526 <== mon.2 v1:172.21.15.97:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f8f8c004b70 con 0x7f8f9813f710 2024-03-25T18:18:25.176 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:25.175+0000 7f8f967fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:25.177 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:25.175+0000 7f8f967fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:18:25.177 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:25.175+0000 7f8f967fc640 10 monclient: dump: 2024-03-25T18:18:25.177 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:18:25.177 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:18:25.177 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:18:25.177 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:18:25.177 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:25.177 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:18:25.177 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:18:25.177 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:18:25.177 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:18:25.177 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:18:25.177 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:25.175+0000 7f8f97fff640 4 set_mon_vals no callback set 2024-03-25T18:18:25.178 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:25.175+0000 7f8f967fc640 1 -- 172.21.15.97:0/4276446526 <== mon.2 v1:172.21.15.97:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2377622137 0 0) 0x7f8f8c004d90 con 0x7f8f9813f710 2024-03-25T18:18:25.178 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:25.176+0000 7f8f9d925640 10 monclient: get_auth_request con 0x7f8f8406de90 auth_method 0 2024-03-25T18:18:25.178 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:25.176+0000 7f8f967fc640 10 client.4307.objecter ms_handle_connect 0x7f8f8406de90 2024-03-25T18:18:25.182 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:25.181+0000 7f8f967fc640 1 -- 172.21.15.97:0/4276446526 <== mon.2 v1:172.21.15.97:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 430711918 0 0) 0x7f8f8c054820 con 0x7f8f9813f710 2024-03-25T18:18:25.182 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:25.181+0000 7f8f967fc640 10 client.4307.objecter ms_dispatch 0x7f8f98000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:18:25.182 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:25.181+0000 7f8f967fc640 3 client.4307.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:18:25.182 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:25.181+0000 7f8f967fc640 3 client.4307.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:18:25.182 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:25.181+0000 7f8f967fc640 20 client.4307.objecter dump_active .. 0 homeless 2024-03-25T18:18:25.183 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:25.181+0000 7f8f967fc640 1 -- 172.21.15.97:0/4276446526 <== mon.2 v1:172.21.15.97:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8f8c050330 con 0x7f8f9813f710 2024-03-25T18:18:25.183 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:25.181+0000 7f8f967fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:25.183 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:25.181+0000 7f8f967fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:18:25.423 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:25.421+0000 7f8f9fb89640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:25.423 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:25.421+0000 7f8f9fb89640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:25.423 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:25.421+0000 7f8f9fb89640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.97:6790/0 2024-03-25T18:18:25.423 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:25.421+0000 7f8f9fb89640 1 -- 172.21.15.97:0/4276446526 --> v1:172.21.15.97:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8f9813df00 con 0x7f8f9813f710 2024-03-25T18:18:25.490 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:25.489+0000 7f8f967fc640 1 -- 172.21.15.97:0/4276446526 <== mon.2 v1:172.21.15.97:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2169427412) 0x7f8f8c052260 con 0x7f8f9813f710 2024-03-25T18:18:25.490 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:25.489+0000 7f8f967fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:25.490 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:25.489+0000 7f8f967fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:18:25.490 INFO:teuthology.misc.health.smithi097.stdout: 2024-03-25T18:18:25.490 INFO:teuthology.misc.health.smithi097.stdout:{"epoch":10,"fsid":"4c3d472c-79b8-486b-9e3d-2738d4306754","created":"2024-03-25T18:16:14.737309+0000","modified":"2024-03-25T18:16:20.357492+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:20.357492+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cb5152b9-262f-45c3-94b8-18d0e81df558","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6daf8793-0838-473d-a7d6-9a0d2894c75d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"136f1493-3b9c-4f43-a2e7-641482d69dae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"397d538c-4f64-4daf-9e8b-662d09f91327","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4f1df08-663f-4bda-93f8-3878ab7c24b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c4b18d70-a3ce-4b15-9b70-eae5525bd761","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8dd4033a-de0e-4f8a-96dc-59f3c1f825b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0077722e-ffa6-4a3d-9891-929d9869b286","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:18:25.492 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:25.491+0000 7f8f9fb89640 10 librados: watch_flush enter 2024-03-25T18:18:25.493 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:25.491+0000 7f8f9fb89640 10 librados: watch_flush exit 2024-03-25T18:18:25.493 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:25.491+0000 7f8f9fb89640 20 client.4307.objecter shutdown clearing up homeless session... 2024-03-25T18:18:25.493 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:25.491+0000 7f8f9fb89640 10 client.4307.objecter successfully canceled tick 2024-03-25T18:18:25.493 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:25.491+0000 7f8f9fb89640 1 -- 172.21.15.97:0/4276446526 >> v1:172.21.15.98:6800/800387573 conn(0x7f8f8406de90 legacy=0x7f8f84070350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:25.493 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:25.491+0000 7f8f9fb89640 10 monclient: shutdown 2024-03-25T18:18:25.493 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:25.491+0000 7f8f9fb89640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:25.493 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:25.491+0000 7f8f9fb89640 1 -- 172.21.15.97:0/4276446526 >> v1:172.21.15.97:6790/0 conn(0x7f8f9813f710 legacy=0x7f8f980ac050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:25.493 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:25.491+0000 7f8f9fb89640 1 -- 172.21.15.97:0/4276446526 shutdown_connections 2024-03-25T18:18:25.494 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:25.491+0000 7f8f9fb89640 1 -- 172.21.15.97:0/4276446526 >> 172.21.15.97:0/4276446526 conn(0x7f8f980b7ea0 msgr2=0x7f8f98074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:25.494 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:25.491+0000 7f8f9fb89640 1 -- 172.21.15.97:0/4276446526 shutdown_connections 2024-03-25T18:18:25.494 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:25.492+0000 7f8f9fb89640 1 -- 172.21.15.97:0/4276446526 wait complete. 2024-03-25T18:18:25.494 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:25.492+0000 7f8f9fb89640 1 librados: shutdown 2024-03-25T18:18:25.504 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:18:32.149 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:02:34.188 34931== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:32.151 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:02:34.190 34931== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:32.186 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:02:34.225 34931== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:32.218 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:02:34.257 34935== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:32.223 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:02:34.262 34935== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:32.235 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:02:34.278 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:32.239 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:02:34.282 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:32.260 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:02:34.299 34935== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:32.265 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:02:34.308 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:32.311 DEBUG:teuthology.orchestra.run.smithi097:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:18:32.425 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.424+0000 7fe6464ba640 10 monclient: get_monmap_and_config 2024-03-25T18:18:32.425 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.424+0000 7fe6464ba640 10 monclient: build_initial_monmap 2024-03-25T18:18:32.425 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.424+0000 7fe6464ba640 10 monclient: monmap: 2024-03-25T18:18:32.425 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:18:32.425 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:18:32.425 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:18:32.425406+0000 2024-03-25T18:18:32.425 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:18:32.425406+0000 2024-03-25T18:18:32.425 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:32.425 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:18:32.425 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:18:32.426 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T18:18:32.426 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-b 2024-03-25T18:18:32.426 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:18:32.426 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.425+0000 7fe6464ba640 1 Processor -- start 2024-03-25T18:18:32.426 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.425+0000 7fe6464ba640 1 -- start start 2024-03-25T18:18:32.426 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.425+0000 7fe6464ba640 10 monclient: init 2024-03-25T18:18:32.427 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.426+0000 7fe6464ba640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:32.427 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.426+0000 7fe6464ba640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:18:32.427 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.426+0000 7fe6464ba640 10 monclient(hunting): picked mon.noname-b con 0x7fe6400b1ef0 addr v1:172.21.15.97:6790/0 2024-03-25T18:18:32.427 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.426+0000 7fe6464ba640 10 monclient(hunting): picked mon.noname-c con 0x7fe6401473a0 addr v1:172.21.15.98:6789/0 2024-03-25T18:18:32.428 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.426+0000 7fe6464ba640 10 monclient(hunting): picked mon.noname-a con 0x7fe640150070 addr v1:172.21.15.97:6789/0 2024-03-25T18:18:32.428 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.426+0000 7fe6464ba640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6400b2570 con 0x7fe640150070 2024-03-25T18:18:32.428 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.426+0000 7fe6464ba640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe640146c20 con 0x7fe6401473a0 2024-03-25T18:18:32.428 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.426+0000 7fe6464ba640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe640146e80 con 0x7fe6400b1ef0 2024-03-25T18:18:32.428 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.426+0000 7fe6464ba640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:32.428 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.426+0000 7fe6464ba640 10 monclient(hunting): authenticate will time out at 7724.702036s 2024-03-25T18:18:32.443 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.442+0000 7fe63f7fe640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fe6401473a0 0x7fe640143b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.97:51546/0 (socket says 172.21.15.97:51546) 2024-03-25T18:18:32.443 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.442+0000 7fe63f7fe640 1 -- 172.21.15.97:0/736453468 learned_addr learned my addr 172.21.15.97:0/736453468 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:18:32.450 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.449+0000 7fe63effd640 1 -- 172.21.15.97:0/736453468 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1718989348 0 0) 0x7fe6400b2570 con 0x7fe640150070 2024-03-25T18:18:32.450 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.449+0000 7fe63effd640 10 monclient(hunting): _negotiate 2024-03-25T18:18:32.451 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.449+0000 7fe63effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:32.451 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.449+0000 7fe63effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:32.451 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.449+0000 7fe63effd640 10 monclient(hunting): my global_id is 4320 2024-03-25T18:18:32.451 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.449+0000 7fe63effd640 1 -- 172.21.15.97:0/736453468 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe628003520 con 0x7fe640150070 2024-03-25T18:18:32.458 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.457+0000 7fe644a57640 1 -- 172.21.15.97:0/736453468 >> v1:172.21.15.97:6789/0 conn(0x7fe640150070 legacy=0x7fe640152460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-03-25T18:18:32.458 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.457+0000 7fe644a57640 1 -- 172.21.15.97:0/736453468 >> v1:172.21.15.97:6789/0 conn(0x7fe640150070 legacy=0x7fe640152460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T18:18:32.458 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.457+0000 7fe644a57640 1 --1- 172.21.15.97:0/736453468 >> v1:172.21.15.97:6789/0 conn(0x7fe640150070 0x7fe640152460 :-1 s=OPENED pgs=12937 cs=1 l=1).handle_message read tag failed 2024-03-25T18:18:32.459 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.457+0000 7fe644a57640 1 --1- 172.21.15.97:0/736453468 >> v1:172.21.15.97:6789/0 conn(0x7fe640150070 0x7fe640152460 :-1 s=OPENED pgs=12937 cs=1 l=1).fault on lossy channel, failing 2024-03-25T18:18:32.459 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.457+0000 7fe63effd640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.97:6789/0 2024-03-25T18:18:32.463 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.462+0000 7fe63effd640 1 -- 172.21.15.97:0/736453468 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3388543793 0 0) 0x7fe640146e80 con 0x7fe6400b1ef0 2024-03-25T18:18:32.463 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.462+0000 7fe63effd640 10 monclient(hunting): _negotiate 2024-03-25T18:18:32.463 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.462+0000 7fe63effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:32.464 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.462+0000 7fe63effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:32.464 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.462+0000 7fe63effd640 10 monclient(hunting): my global_id is 4310 2024-03-25T18:18:32.464 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.462+0000 7fe63effd640 1 -- 172.21.15.97:0/736453468 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6400b2570 con 0x7fe6400b1ef0 2024-03-25T18:18:32.464 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.463+0000 7fe63effd640 1 -- 172.21.15.97:0/736453468 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2097801333 0 0) 0x7fe640146c20 con 0x7fe6401473a0 2024-03-25T18:18:32.464 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.463+0000 7fe63effd640 10 monclient(hunting): _negotiate 2024-03-25T18:18:32.464 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.463+0000 7fe63effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:32.465 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.463+0000 7fe63effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:32.465 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.463+0000 7fe63effd640 10 monclient(hunting): my global_id is 4315 2024-03-25T18:18:32.465 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.463+0000 7fe63effd640 1 -- 172.21.15.97:0/736453468 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe640146e80 con 0x7fe6401473a0 2024-03-25T18:18:32.496 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.495+0000 7fe63effd640 1 -- 172.21.15.97:0/736453468 <== mon.2 v1:172.21.15.97:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4043375548 0 0) 0x7fe6400b2570 con 0x7fe6400b1ef0 2024-03-25T18:18:32.496 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.495+0000 7fe63effd640 1 -- 172.21.15.97:0/736453468 >> v1:172.21.15.98:6789/0 conn(0x7fe6401473a0 legacy=0x7fe640143b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:32.497 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.495+0000 7fe63effd640 1 -- 172.21.15.97:0/736453468 >> v1:172.21.15.97:6789/0 conn(0x7fe640150070 legacy=0x7fe640152460 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T18:18:32.497 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.495+0000 7fe63effd640 10 monclient: _finish_hunting 0 2024-03-25T18:18:32.497 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.495+0000 7fe63effd640 1 monclient: found mon.noname-b 2024-03-25T18:18:32.497 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.495+0000 7fe63effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:32.497 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.495+0000 7fe63effd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.97:6790/0 2024-03-25T18:18:32.497 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.495+0000 7fe63effd640 1 -- 172.21.15.97:0/736453468 --> v1:172.21.15.97:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe64006e1b0 con 0x7fe6400b1ef0 2024-03-25T18:18:32.497 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.495+0000 7fe63effd640 10 monclient: _finish_auth 0 2024-03-25T18:18:32.497 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.495+0000 7fe63effd640 10 monclient: _check_auth_tickets 2024-03-25T18:18:32.497 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.495+0000 7fe63effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:32.497 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.495+0000 7fe63effd640 1 -- 172.21.15.97:0/736453468 <== mon.2 v1:172.21.15.97:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7fe6380019b0 con 0x7fe6400b1ef0 2024-03-25T18:18:32.497 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.495+0000 7fe6464ba640 5 monclient: authenticate success, global_id 4310 2024-03-25T18:18:32.497 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.495+0000 7fe6464ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:18:32.497 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.495+0000 7fe63effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:32.497 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.495+0000 7fe63effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:18:32.498 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.495+0000 7fe63effd640 10 monclient: dump: 2024-03-25T18:18:32.498 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:18:32.498 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:18:32.498 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:18:32.498 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:18:32.498 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:32.498 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:18:32.498 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:18:32.498 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:18:32.498 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:18:32.498 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:18:32.498 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.495+0000 7fe6464ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:18:32.543 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.542+0000 7fe63effd640 1 -- 172.21.15.97:0/736453468 <== mon.2 v1:172.21.15.97:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe638001b10 con 0x7fe6400b1ef0 2024-03-25T18:18:32.543 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.542+0000 7fe63effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:32.544 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.542+0000 7fe63effd640 1 -- 172.21.15.97:0/736453468 <== mon.2 v1:172.21.15.97:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7fe638001f20 con 0x7fe6400b1ef0 2024-03-25T18:18:32.544 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.542+0000 7fe6464ba640 10 monclient: get_monmap_and_config success 2024-03-25T18:18:32.544 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.542+0000 7fe6464ba640 4 set_mon_vals no callback set 2024-03-25T18:18:32.544 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.542+0000 7fe63effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:32.544 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.542+0000 7fe63effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:18:32.544 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.542+0000 7fe63effd640 10 monclient: dump: 2024-03-25T18:18:32.544 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:18:32.544 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:18:32.544 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:18:32.544 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:18:32.544 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:32.544 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:18:32.544 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:18:32.545 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:18:32.545 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:18:32.545 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:18:32.547 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.543+0000 7fe6464ba640 10 monclient: shutdown 2024-03-25T18:18:32.547 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.543+0000 7fe6464ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:32.547 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.543+0000 7fe6464ba640 1 -- 172.21.15.97:0/736453468 >> v1:172.21.15.97:6790/0 conn(0x7fe6400b1ef0 legacy=0x7fe6400b28a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:32.547 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.543+0000 7fe6464ba640 1 -- 172.21.15.97:0/736453468 shutdown_connections 2024-03-25T18:18:32.547 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.543+0000 7fe6464ba640 1 -- 172.21.15.97:0/736453468 >> 172.21.15.97:0/736453468 conn(0x7fe6400b7ea0 msgr2=0x7fe6400b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:32.547 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.543+0000 7fe6464ba640 1 -- 172.21.15.97:0/736453468 shutdown_connections 2024-03-25T18:18:32.547 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.543+0000 7fe6464ba640 1 -- 172.21.15.97:0/736453468 wait complete. 2024-03-25T18:18:32.547 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.544+0000 7fe6464ba640 10 monclient: build_initial_monmap 2024-03-25T18:18:32.547 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.544+0000 7fe6464ba640 10 monclient: monmap: 2024-03-25T18:18:32.548 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:18:32.548 INFO:teuthology.misc.health.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:18:32.548 INFO:teuthology.misc.health.smithi097.stderr:last_changed 0.000000 2024-03-25T18:18:32.548 INFO:teuthology.misc.health.smithi097.stderr:created 0.000000 2024-03-25T18:18:32.548 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:32.548 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 1 2024-03-25T18:18:32.548 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:18:32.548 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T18:18:32.548 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-c 2024-03-25T18:18:32.548 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:18:32.548 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.544+0000 7fe6464ba640 1 librados: starting msgr at 2024-03-25T18:18:32.548 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.544+0000 7fe6464ba640 1 librados: starting objecter 2024-03-25T18:18:32.548 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.544+0000 7fe6464ba640 1 Processor -- start 2024-03-25T18:18:32.548 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.544+0000 7fe6464ba640 1 -- start start 2024-03-25T18:18:32.549 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.544+0000 7fe6464ba640 1 librados: setting wanted keys 2024-03-25T18:18:32.549 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.544+0000 7fe6464ba640 1 librados: calling monclient init 2024-03-25T18:18:32.549 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.544+0000 7fe6464ba640 10 monclient: init 2024-03-25T18:18:32.549 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.545+0000 7fe6464ba640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:32.549 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.545+0000 7fe6464ba640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:18:32.549 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.545+0000 7fe6464ba640 10 monclient(hunting): picked mon.noname-a con 0x7fe6400b1ef0 addr v1:172.21.15.97:6789/0 2024-03-25T18:18:32.549 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.545+0000 7fe6464ba640 10 monclient(hunting): picked mon.noname-b con 0x7fe6401473a0 addr v1:172.21.15.98:6789/0 2024-03-25T18:18:32.549 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.545+0000 7fe6464ba640 10 monclient(hunting): picked mon.noname-c con 0x7fe640150070 addr v1:172.21.15.97:6790/0 2024-03-25T18:18:32.549 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.545+0000 7fe6464ba640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6400b32c0 con 0x7fe6400b1ef0 2024-03-25T18:18:32.549 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.545+0000 7fe6464ba640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe640144e90 con 0x7fe6401473a0 2024-03-25T18:18:32.549 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.545+0000 7fe6464ba640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe640146710 con 0x7fe640150070 2024-03-25T18:18:32.549 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.545+0000 7fe6464ba640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:32.549 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.545+0000 7fe6464ba640 10 monclient(hunting): authenticate will time out at 7724.821787s 2024-03-25T18:18:32.552 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.551+0000 7fe644a57640 1 --1- >> v1:172.21.15.97:6790/0 conn(0x7fe640150070 0x7fe6401e6fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.97:6790/0 says I am v1:172.21.15.97:36906/0 (socket says 172.21.15.97:36906) 2024-03-25T18:18:32.552 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.551+0000 7fe644a57640 1 -- 172.21.15.97:0/3512659963 learned_addr learned my addr 172.21.15.97:0/3512659963 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:18:32.554 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.553+0000 7fe63cff9640 10 client.?.objecter ms_handle_connect 0x7fe640150070 2024-03-25T18:18:32.554 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.553+0000 7fe63cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:32.562 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.561+0000 7fe63cff9640 10 client.?.objecter ms_handle_connect 0x7fe6400b1ef0 2024-03-25T18:18:32.563 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.561+0000 7fe63cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:32.565 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.564+0000 7fe63cff9640 1 -- 172.21.15.97:0/3512659963 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3726058206 0 0) 0x7fe640146710 con 0x7fe640150070 2024-03-25T18:18:32.565 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.564+0000 7fe63cff9640 10 monclient(hunting): _negotiate 2024-03-25T18:18:32.565 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.564+0000 7fe63cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:32.565 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.564+0000 7fe63cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:32.565 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.564+0000 7fe63cff9640 10 monclient(hunting): my global_id is 4313 2024-03-25T18:18:32.565 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.564+0000 7fe63cff9640 1 -- 172.21.15.97:0/3512659963 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe62c004eb0 con 0x7fe640150070 2024-03-25T18:18:32.569 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.568+0000 7fe63cff9640 10 client.?.objecter ms_handle_connect 0x7fe6401473a0 2024-03-25T18:18:32.569 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.568+0000 7fe63cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:32.574 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.573+0000 7fe63cff9640 1 -- 172.21.15.97:0/3512659963 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2431467352 0 0) 0x7fe6400b32c0 con 0x7fe6400b1ef0 2024-03-25T18:18:32.574 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.573+0000 7fe63cff9640 10 monclient(hunting): _negotiate 2024-03-25T18:18:32.574 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.573+0000 7fe63cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:32.574 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.573+0000 7fe63cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:32.574 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.573+0000 7fe63cff9640 10 monclient(hunting): my global_id is 4323 2024-03-25T18:18:32.574 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.573+0000 7fe63cff9640 1 -- 172.21.15.97:0/3512659963 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe640146710 con 0x7fe6400b1ef0 2024-03-25T18:18:32.576 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.575+0000 7fe63cff9640 1 -- 172.21.15.97:0/3512659963 <== mon.2 v1:172.21.15.97:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4078275895 0 0) 0x7fe62c004eb0 con 0x7fe640150070 2024-03-25T18:18:32.576 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.575+0000 7fe63cff9640 1 -- 172.21.15.97:0/3512659963 --> v1:172.21.15.97:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe6400b32c0 con 0x7fe640150070 2024-03-25T18:18:32.576 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.575+0000 7fe63cff9640 1 -- 172.21.15.97:0/3512659963 <== mon.2 v1:172.21.15.97:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7fe630001900 con 0x7fe640150070 2024-03-25T18:18:32.576 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.575+0000 7fe63cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:32.576 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.575+0000 7fe63cff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:18:32.577 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.575+0000 7fe63cff9640 10 monclient(hunting): dump: 2024-03-25T18:18:32.577 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:18:32.577 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:18:32.577 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:18:32.577 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:18:32.577 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:32.577 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:18:32.577 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:18:32.577 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:18:32.577 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:18:32.577 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:18:32.577 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.575+0000 7fe63cff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:18:32.581 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.580+0000 7fe63cff9640 1 -- 172.21.15.97:0/3512659963 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3439225651 0 0) 0x7fe640144e90 con 0x7fe6401473a0 2024-03-25T18:18:32.581 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.580+0000 7fe63cff9640 10 monclient(hunting): _negotiate 2024-03-25T18:18:32.582 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.580+0000 7fe63cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:32.582 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.580+0000 7fe63cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:32.582 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.580+0000 7fe63cff9640 10 monclient(hunting): my global_id is 4318 2024-03-25T18:18:32.582 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.580+0000 7fe63cff9640 1 -- 172.21.15.97:0/3512659963 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe62c004eb0 con 0x7fe6401473a0 2024-03-25T18:18:32.585 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.583+0000 7fe63cff9640 1 -- 172.21.15.97:0/3512659963 <== mon.2 v1:172.21.15.97:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1760237323 0 0) 0x7fe6400b32c0 con 0x7fe640150070 2024-03-25T18:18:32.585 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.583+0000 7fe63cff9640 1 -- 172.21.15.97:0/3512659963 >> v1:172.21.15.98:6789/0 conn(0x7fe6401473a0 legacy=0x7fe6401453b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:32.585 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.583+0000 7fe63cff9640 1 -- 172.21.15.97:0/3512659963 >> v1:172.21.15.97:6789/0 conn(0x7fe6400b1ef0 legacy=0x7fe64013a070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:32.585 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.583+0000 7fe63cff9640 10 monclient: _finish_hunting 0 2024-03-25T18:18:32.585 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.583+0000 7fe63cff9640 1 monclient: found mon.c 2024-03-25T18:18:32.585 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.583+0000 7fe63cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:32.585 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.583+0000 7fe63cff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.97:6790/0 2024-03-25T18:18:32.585 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.583+0000 7fe63cff9640 1 -- 172.21.15.97:0/3512659963 --> v1:172.21.15.97:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe640144950 con 0x7fe640150070 2024-03-25T18:18:32.585 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.583+0000 7fe63cff9640 10 monclient: _finish_auth 0 2024-03-25T18:18:32.586 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.583+0000 7fe63cff9640 10 monclient: _check_auth_tickets 2024-03-25T18:18:32.586 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.583+0000 7fe63cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:32.586 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.583+0000 7fe6464ba640 5 monclient: authenticate success, global_id 4313 2024-03-25T18:18:32.586 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.583+0000 7fe6464ba640 10 monclient: _renew_subs 2024-03-25T18:18:32.586 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.583+0000 7fe6464ba640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.97:6790/0 2024-03-25T18:18:32.586 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.583+0000 7fe6464ba640 1 -- 172.21.15.97:0/3512659963 --> v1:172.21.15.97:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6401df8a0 con 0x7fe640150070 2024-03-25T18:18:32.586 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.583+0000 7fe6464ba640 10 client.4313.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:18:32.586 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.583+0000 7fe6464ba640 10 monclient: _renew_subs 2024-03-25T18:18:32.586 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.583+0000 7fe6464ba640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.97:6790/0 2024-03-25T18:18:32.586 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.583+0000 7fe6464ba640 1 -- 172.21.15.97:0/3512659963 --> v1:172.21.15.97:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6401df060 con 0x7fe640150070 2024-03-25T18:18:32.586 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.583+0000 7fe6464ba640 1 librados: init done 2024-03-25T18:18:32.587 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.584+0000 7fe6464ba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:32.587 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.584+0000 7fe6464ba640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:32.587 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.584+0000 7fe6464ba640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.97:6790/0 2024-03-25T18:18:32.587 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.584+0000 7fe6464ba640 1 -- 172.21.15.97:0/3512659963 --> v1:172.21.15.97:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe640145020 con 0x7fe640150070 2024-03-25T18:18:32.890 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.889+0000 7fe63cff9640 1 -- 172.21.15.97:0/3512659963 <== mon.2 v1:172.21.15.97:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe6300035f0 con 0x7fe640150070 2024-03-25T18:18:32.890 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.889+0000 7fe63cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:32.890 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.889+0000 7fe63cff9640 1 -- 172.21.15.97:0/3512659963 <== mon.2 v1:172.21.15.97:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7fe630003980 con 0x7fe640150070 2024-03-25T18:18:32.890 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.889+0000 7fe63cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:32.890 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.889+0000 7fe63cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:18:32.890 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.889+0000 7fe63cff9640 10 monclient: dump: 2024-03-25T18:18:32.891 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:18:32.891 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:18:32.891 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:18:32.891 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:18:32.891 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:32.891 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:18:32.891 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:18:32.891 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:18:32.891 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:18:32.891 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:18:32.891 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.889+0000 7fe63e7fc640 4 set_mon_vals no callback set 2024-03-25T18:18:32.892 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.890+0000 7fe63cff9640 1 -- 172.21.15.97:0/3512659963 <== mon.2 v1:172.21.15.97:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2377622137 0 0) 0x7fe630003b60 con 0x7fe640150070 2024-03-25T18:18:32.892 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.890+0000 7fe63ffff640 10 monclient: get_auth_request con 0x7fe62c06dda0 auth_method 0 2024-03-25T18:18:32.892 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.891+0000 7fe63cff9640 10 client.4313.objecter ms_handle_connect 0x7fe62c06dda0 2024-03-25T18:18:32.896 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.894+0000 7fe63cff9640 1 -- 172.21.15.97:0/3512659963 <== mon.2 v1:172.21.15.97:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 430711918 0 0) 0x7fe630054800 con 0x7fe640150070 2024-03-25T18:18:32.896 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.894+0000 7fe63cff9640 10 client.4313.objecter ms_dispatch 0x7fe640000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:18:32.896 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.894+0000 7fe63cff9640 3 client.4313.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:18:32.896 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.894+0000 7fe63cff9640 3 client.4313.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:18:32.896 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.894+0000 7fe63cff9640 20 client.4313.objecter dump_active .. 0 homeless 2024-03-25T18:18:32.896 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.894+0000 7fe63cff9640 1 -- 172.21.15.97:0/3512659963 <== mon.2 v1:172.21.15.97:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe630050310 con 0x7fe640150070 2024-03-25T18:18:32.896 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.894+0000 7fe63cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:32.896 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:32.894+0000 7fe63cff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:18:33.128 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:33.127+0000 7fe6464ba640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:33.128 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:33.127+0000 7fe6464ba640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:33.128 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:33.127+0000 7fe6464ba640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.97:6790/0 2024-03-25T18:18:33.128 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:33.127+0000 7fe6464ba640 1 -- 172.21.15.97:0/3512659963 --> v1:172.21.15.97:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe6400627b0 con 0x7fe640150070 2024-03-25T18:18:33.157 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:33.155+0000 7fe63cff9640 1 -- 172.21.15.97:0/3512659963 <== mon.2 v1:172.21.15.97:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2169427412) 0x7fe630052240 con 0x7fe640150070 2024-03-25T18:18:33.157 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:33.155+0000 7fe63cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:33.157 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:33.155+0000 7fe63cff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:18:33.157 INFO:teuthology.misc.health.smithi097.stdout: 2024-03-25T18:18:33.157 INFO:teuthology.misc.health.smithi097.stdout:{"epoch":10,"fsid":"4c3d472c-79b8-486b-9e3d-2738d4306754","created":"2024-03-25T18:16:14.737309+0000","modified":"2024-03-25T18:16:20.357492+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:20.357492+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cb5152b9-262f-45c3-94b8-18d0e81df558","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6daf8793-0838-473d-a7d6-9a0d2894c75d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"136f1493-3b9c-4f43-a2e7-641482d69dae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"397d538c-4f64-4daf-9e8b-662d09f91327","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4f1df08-663f-4bda-93f8-3878ab7c24b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c4b18d70-a3ce-4b15-9b70-eae5525bd761","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8dd4033a-de0e-4f8a-96dc-59f3c1f825b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0077722e-ffa6-4a3d-9891-929d9869b286","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:18:33.160 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:33.158+0000 7fe6464ba640 10 librados: watch_flush enter 2024-03-25T18:18:33.160 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:33.158+0000 7fe6464ba640 10 librados: watch_flush exit 2024-03-25T18:18:33.160 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:33.158+0000 7fe6464ba640 20 client.4313.objecter shutdown clearing up homeless session... 2024-03-25T18:18:33.160 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:33.158+0000 7fe6464ba640 10 client.4313.objecter successfully canceled tick 2024-03-25T18:18:33.160 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:33.158+0000 7fe6464ba640 1 -- 172.21.15.97:0/3512659963 >> v1:172.21.15.98:6800/800387573 conn(0x7fe62c06dda0 legacy=0x7fe62c070260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:33.160 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:33.158+0000 7fe6464ba640 10 monclient: shutdown 2024-03-25T18:18:33.160 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:33.158+0000 7fe6464ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:33.160 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:33.158+0000 7fe6464ba640 1 -- 172.21.15.97:0/3512659963 >> v1:172.21.15.97:6790/0 conn(0x7fe640150070 legacy=0x7fe6401e6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:33.160 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:33.158+0000 7fe6464ba640 1 -- 172.21.15.97:0/3512659963 shutdown_connections 2024-03-25T18:18:33.160 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:33.158+0000 7fe6464ba640 1 -- 172.21.15.97:0/3512659963 >> 172.21.15.97:0/3512659963 conn(0x7fe6400b7ea0 msgr2=0x7fe64013d290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:33.160 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:33.158+0000 7fe6464ba640 1 -- 172.21.15.97:0/3512659963 shutdown_connections 2024-03-25T18:18:33.160 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:33.158+0000 7fe6464ba640 1 -- 172.21.15.97:0/3512659963 wait complete. 2024-03-25T18:18:33.160 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:33.158+0000 7fe6464ba640 1 librados: shutdown 2024-03-25T18:18:33.172 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:18:39.979 DEBUG:teuthology.orchestra.run.smithi097:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:18:40.094 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.094+0000 7fe8d7d4b640 10 monclient: get_monmap_and_config 2024-03-25T18:18:40.094 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.094+0000 7fe8d7d4b640 10 monclient: build_initial_monmap 2024-03-25T18:18:40.094 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.094+0000 7fe8d7d4b640 10 monclient: monmap: 2024-03-25T18:18:40.094 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:18:40.094 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:18:40.095 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:18:40.094489+0000 2024-03-25T18:18:40.095 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:18:40.094489+0000 2024-03-25T18:18:40.095 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:40.095 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:18:40.095 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:18:40.095 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T18:18:40.095 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-b 2024-03-25T18:18:40.095 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:18:40.096 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.095+0000 7fe8d7d4b640 1 Processor -- start 2024-03-25T18:18:40.096 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.095+0000 7fe8d7d4b640 1 -- start start 2024-03-25T18:18:40.096 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.095+0000 7fe8d7d4b640 10 monclient: init 2024-03-25T18:18:40.096 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.096+0000 7fe8d7d4b640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:40.097 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.096+0000 7fe8d7d4b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:18:40.097 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.096+0000 7fe8d7d4b640 10 monclient(hunting): picked mon.noname-b con 0x7fe8d00b4d60 addr v1:172.21.15.97:6790/0 2024-03-25T18:18:40.097 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.096+0000 7fe8d7d4b640 10 monclient(hunting): picked mon.noname-c con 0x7fe8d00b1730 addr v1:172.21.15.98:6789/0 2024-03-25T18:18:40.097 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.096+0000 7fe8d7d4b640 10 monclient(hunting): picked mon.noname-a con 0x7fe8d014be90 addr v1:172.21.15.97:6789/0 2024-03-25T18:18:40.097 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.096+0000 7fe8d7d4b640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8d00af940 con 0x7fe8d014be90 2024-03-25T18:18:40.097 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.096+0000 7fe8d7d4b640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8d00b0a70 con 0x7fe8d00b1730 2024-03-25T18:18:40.097 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.096+0000 7fe8d7d4b640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8d00b0d50 con 0x7fe8d00b4d60 2024-03-25T18:18:40.097 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.096+0000 7fe8d7d4b640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:40.097 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.096+0000 7fe8d7d4b640 10 monclient(hunting): authenticate will time out at 7732.371206s 2024-03-25T18:18:40.099 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.099+0000 7fe8d5ae7640 1 --1- >> v1:172.21.15.97:6790/0 conn(0x7fe8d00b4d60 0x7fe8d013f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6790/0 says I am v1:172.21.15.97:41830/0 (socket says 172.21.15.97:41830) 2024-03-25T18:18:40.099 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.099+0000 7fe8d5ae7640 1 -- 172.21.15.97:0/3561540819 learned_addr learned my addr 172.21.15.97:0/3561540819 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:18:40.108 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.108+0000 7fe8d4ae5640 1 -- 172.21.15.97:0/3561540819 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2595273310 0 0) 0x7fe8d00b0d50 con 0x7fe8d00b4d60 2024-03-25T18:18:40.108 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.108+0000 7fe8d4ae5640 10 monclient(hunting): _negotiate 2024-03-25T18:18:40.108 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.108+0000 7fe8d4ae5640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:40.108 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.108+0000 7fe8d4ae5640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:40.108 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.108+0000 7fe8d4ae5640 10 monclient(hunting): my global_id is 4316 2024-03-25T18:18:40.108 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.108+0000 7fe8d4ae5640 1 -- 172.21.15.97:0/3561540819 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8bc003520 con 0x7fe8d00b4d60 2024-03-25T18:18:40.112 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.111+0000 7fe8d4ae5640 1 -- 172.21.15.97:0/3561540819 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 873420698 0 0) 0x7fe8d00b0a70 con 0x7fe8d00b1730 2024-03-25T18:18:40.112 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.111+0000 7fe8d4ae5640 10 monclient(hunting): _negotiate 2024-03-25T18:18:40.112 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.111+0000 7fe8d4ae5640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:40.112 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.111+0000 7fe8d4ae5640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:40.112 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.111+0000 7fe8d4ae5640 10 monclient(hunting): my global_id is 4321 2024-03-25T18:18:40.112 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.112+0000 7fe8d4ae5640 1 -- 172.21.15.97:0/3561540819 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8d00b0d50 con 0x7fe8d00b1730 2024-03-25T18:18:40.115 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.115+0000 7fe8d4ae5640 1 -- 172.21.15.97:0/3561540819 <== mon.2 v1:172.21.15.97:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2552952005 0 0) 0x7fe8bc003520 con 0x7fe8d00b4d60 2024-03-25T18:18:40.115 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.115+0000 7fe8d4ae5640 1 -- 172.21.15.97:0/3561540819 >> v1:172.21.15.98:6789/0 conn(0x7fe8d00b1730 legacy=0x7fe8d00b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:40.115 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.115+0000 7fe8d4ae5640 1 -- 172.21.15.97:0/3561540819 >> v1:172.21.15.97:6789/0 conn(0x7fe8d014be90 legacy=0x7fe8d014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:18:40.116 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.115+0000 7fe8d4ae5640 10 monclient: _finish_hunting 0 2024-03-25T18:18:40.116 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.115+0000 7fe8d4ae5640 1 monclient: found mon.noname-b 2024-03-25T18:18:40.116 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.115+0000 7fe8d4ae5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:40.116 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.115+0000 7fe8d4ae5640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.97:6790/0 2024-03-25T18:18:40.116 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.115+0000 7fe8d4ae5640 1 -- 172.21.15.97:0/3561540819 --> v1:172.21.15.97:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8d00b1030 con 0x7fe8d00b4d60 2024-03-25T18:18:40.116 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.115+0000 7fe8d4ae5640 10 monclient: _finish_auth 0 2024-03-25T18:18:40.116 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.115+0000 7fe8d4ae5640 10 monclient: _check_auth_tickets 2024-03-25T18:18:40.116 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.115+0000 7fe8d4ae5640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:40.116 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.115+0000 7fe8d4ae5640 1 -- 172.21.15.97:0/3561540819 <== mon.2 v1:172.21.15.97:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7fe8cc001690 con 0x7fe8d00b4d60 2024-03-25T18:18:40.116 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.115+0000 7fe8d4ae5640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:40.116 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.115+0000 7fe8d4ae5640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:18:40.116 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.115+0000 7fe8d4ae5640 10 monclient: dump: 2024-03-25T18:18:40.116 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:18:40.116 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:18:40.117 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:18:40.117 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:18:40.117 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:40.117 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:18:40.117 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:18:40.117 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:18:40.117 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:18:40.117 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:18:40.117 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.115+0000 7fe8d7d4b640 5 monclient: authenticate success, global_id 4316 2024-03-25T18:18:40.117 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.115+0000 7fe8d7d4b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:18:40.126 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.125+0000 7fe8d4ae5640 1 -- 172.21.15.97:0/3561540819 <== mon.2 v1:172.21.15.97:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe8cc0017f0 con 0x7fe8d00b4d60 2024-03-25T18:18:40.126 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.125+0000 7fe8d4ae5640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:40.126 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.125+0000 7fe8d4ae5640 1 -- 172.21.15.97:0/3561540819 <== mon.2 v1:172.21.15.97:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7fe8cc001c00 con 0x7fe8d00b4d60 2024-03-25T18:18:40.126 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.125+0000 7fe8d4ae5640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:40.126 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.125+0000 7fe8d4ae5640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:18:40.126 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.125+0000 7fe8d4ae5640 10 monclient: dump: 2024-03-25T18:18:40.126 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:18:40.126 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:18:40.126 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:18:40.126 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:18:40.126 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:40.127 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:18:40.127 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:18:40.127 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:18:40.127 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:18:40.127 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:18:40.127 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.125+0000 7fe8d7d4b640 10 monclient: get_monmap_and_config success 2024-03-25T18:18:40.127 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.125+0000 7fe8d7d4b640 4 set_mon_vals no callback set 2024-03-25T18:18:40.128 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.126+0000 7fe8d7d4b640 10 monclient: shutdown 2024-03-25T18:18:40.128 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.126+0000 7fe8d7d4b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:40.128 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.126+0000 7fe8d7d4b640 1 -- 172.21.15.97:0/3561540819 >> v1:172.21.15.97:6790/0 conn(0x7fe8d00b4d60 legacy=0x7fe8d013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:40.128 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.126+0000 7fe8d7d4b640 1 -- 172.21.15.97:0/3561540819 shutdown_connections 2024-03-25T18:18:40.128 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.126+0000 7fe8d7d4b640 1 -- 172.21.15.97:0/3561540819 >> 172.21.15.97:0/3561540819 conn(0x7fe8d00617a0 msgr2=0x7fe8d0061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:40.129 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.126+0000 7fe8d7d4b640 1 -- 172.21.15.97:0/3561540819 shutdown_connections 2024-03-25T18:18:40.129 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.126+0000 7fe8d7d4b640 1 -- 172.21.15.97:0/3561540819 wait complete. 2024-03-25T18:18:40.129 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.127+0000 7fe8d7d4b640 10 monclient: build_initial_monmap 2024-03-25T18:18:40.129 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.127+0000 7fe8d7d4b640 10 monclient: monmap: 2024-03-25T18:18:40.129 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:18:40.129 INFO:teuthology.misc.health.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:18:40.129 INFO:teuthology.misc.health.smithi097.stderr:last_changed 0.000000 2024-03-25T18:18:40.129 INFO:teuthology.misc.health.smithi097.stderr:created 0.000000 2024-03-25T18:18:40.129 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:40.129 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 1 2024-03-25T18:18:40.129 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:18:40.129 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T18:18:40.129 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-c 2024-03-25T18:18:40.129 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:18:40.129 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.128+0000 7fe8d7d4b640 1 librados: starting msgr at 2024-03-25T18:18:40.130 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.128+0000 7fe8d7d4b640 1 librados: starting objecter 2024-03-25T18:18:40.130 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.128+0000 7fe8d7d4b640 1 Processor -- start 2024-03-25T18:18:40.130 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.128+0000 7fe8d7d4b640 1 -- start start 2024-03-25T18:18:40.130 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.128+0000 7fe8d7d4b640 1 librados: setting wanted keys 2024-03-25T18:18:40.130 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.128+0000 7fe8d7d4b640 1 librados: calling monclient init 2024-03-25T18:18:40.130 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.128+0000 7fe8d7d4b640 10 monclient: init 2024-03-25T18:18:40.131 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.128+0000 7fe8d7d4b640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:40.131 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.128+0000 7fe8d7d4b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:18:40.131 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.128+0000 7fe8d7d4b640 10 monclient(hunting): picked mon.noname-c con 0x7fe8d00b1730 addr v1:172.21.15.97:6790/0 2024-03-25T18:18:40.131 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.128+0000 7fe8d7d4b640 10 monclient(hunting): picked mon.noname-a con 0x7fe8d00b4d60 addr v1:172.21.15.97:6789/0 2024-03-25T18:18:40.131 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.129+0000 7fe8d7d4b640 10 monclient(hunting): picked mon.noname-b con 0x7fe8d014be90 addr v1:172.21.15.98:6789/0 2024-03-25T18:18:40.131 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.129+0000 7fe8d7d4b640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8d0142880 con 0x7fe8d00b4d60 2024-03-25T18:18:40.131 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.129+0000 7fe8d7d4b640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8d01d84b0 con 0x7fe8d014be90 2024-03-25T18:18:40.131 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.129+0000 7fe8d7d4b640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8d0140e30 con 0x7fe8d00b1730 2024-03-25T18:18:40.131 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.129+0000 7fe8d7d4b640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:40.131 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.129+0000 7fe8d7d4b640 10 monclient(hunting): authenticate will time out at 7732.403933s 2024-03-25T18:18:40.140 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.140+0000 7fe8d52e6640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fe8d00b4d60 0x7fe8d0140380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:32832/0 (socket says 172.21.15.97:32832) 2024-03-25T18:18:40.140 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.140+0000 7fe8d52e6640 1 -- 172.21.15.97:0/3735760359 learned_addr learned my addr 172.21.15.97:0/3735760359 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:18:40.141 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.141+0000 7fe8c67fc640 10 client.?.objecter ms_handle_connect 0x7fe8d00b4d60 2024-03-25T18:18:40.141 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.141+0000 7fe8c67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:40.142 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.142+0000 7fe8c67fc640 10 client.?.objecter ms_handle_connect 0x7fe8d00b1730 2024-03-25T18:18:40.142 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.142+0000 7fe8c67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:40.154 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.153+0000 7fe8c67fc640 1 -- 172.21.15.97:0/3735760359 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 45687551 0 0) 0x7fe8d0140e30 con 0x7fe8d00b1730 2024-03-25T18:18:40.154 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.153+0000 7fe8c67fc640 10 monclient(hunting): _negotiate 2024-03-25T18:18:40.154 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.153+0000 7fe8c67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:40.154 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.153+0000 7fe8c67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:40.154 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.153+0000 7fe8c67fc640 10 monclient(hunting): my global_id is 4319 2024-03-25T18:18:40.154 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.153+0000 7fe8c67fc640 1 -- 172.21.15.97:0/3735760359 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8b4004eb0 con 0x7fe8d00b1730 2024-03-25T18:18:40.154 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.153+0000 7fe8c67fc640 1 -- 172.21.15.97:0/3735760359 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 513449648 0 0) 0x7fe8d0142880 con 0x7fe8d00b4d60 2024-03-25T18:18:40.154 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.154+0000 7fe8c67fc640 10 monclient(hunting): _negotiate 2024-03-25T18:18:40.154 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.154+0000 7fe8c67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:40.154 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.154+0000 7fe8c67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:40.154 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.154+0000 7fe8c67fc640 10 monclient(hunting): my global_id is 4326 2024-03-25T18:18:40.154 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.154+0000 7fe8c67fc640 1 -- 172.21.15.97:0/3735760359 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8d0140e30 con 0x7fe8d00b4d60 2024-03-25T18:18:40.161 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.161+0000 7fe8c67fc640 10 client.?.objecter ms_handle_connect 0x7fe8d014be90 2024-03-25T18:18:40.161 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.161+0000 7fe8c67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:40.172 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.171+0000 7fe8c67fc640 1 -- 172.21.15.97:0/3735760359 <== mon.2 v1:172.21.15.97:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 301394624 0 0) 0x7fe8b4004eb0 con 0x7fe8d00b1730 2024-03-25T18:18:40.172 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.172+0000 7fe8c67fc640 1 -- 172.21.15.97:0/3735760359 --> v1:172.21.15.97:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe8d0142880 con 0x7fe8d00b1730 2024-03-25T18:18:40.172 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.172+0000 7fe8c67fc640 1 -- 172.21.15.97:0/3735760359 <== mon.2 v1:172.21.15.97:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7fe8cc001bc0 con 0x7fe8d00b1730 2024-03-25T18:18:40.172 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.172+0000 7fe8c67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:40.172 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.172+0000 7fe8c67fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:18:40.173 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.172+0000 7fe8c67fc640 10 monclient(hunting): dump: 2024-03-25T18:18:40.173 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:18:40.173 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:18:40.173 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:18:40.173 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:18:40.173 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:40.173 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:18:40.173 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:18:40.173 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:18:40.173 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:18:40.173 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:18:40.173 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.172+0000 7fe8c67fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:18:40.173 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.172+0000 7fe8c67fc640 1 -- 172.21.15.97:0/3735760359 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1007901462 0 0) 0x7fe8d0140e30 con 0x7fe8d00b4d60 2024-03-25T18:18:40.173 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.172+0000 7fe8c67fc640 1 -- 172.21.15.97:0/3735760359 --> v1:172.21.15.97:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe8b4004eb0 con 0x7fe8d00b4d60 2024-03-25T18:18:40.173 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.172+0000 7fe8c67fc640 1 -- 172.21.15.97:0/3735760359 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7fe8c00019f0 con 0x7fe8d00b4d60 2024-03-25T18:18:40.174 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.172+0000 7fe8c67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:40.174 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.172+0000 7fe8c67fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:18:40.174 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.172+0000 7fe8c67fc640 10 monclient(hunting): dump: 2024-03-25T18:18:40.174 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:18:40.174 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:18:40.174 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:18:40.174 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:18:40.174 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:40.174 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:18:40.174 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:18:40.174 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:18:40.174 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:18:40.174 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:18:40.175 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.175+0000 7fe8c67fc640 1 -- 172.21.15.97:0/3735760359 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2584752887 0 0) 0x7fe8d01d84b0 con 0x7fe8d014be90 2024-03-25T18:18:40.175 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.175+0000 7fe8c67fc640 10 monclient(hunting): _negotiate 2024-03-25T18:18:40.175 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.175+0000 7fe8c67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:40.175 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.175+0000 7fe8c67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:40.175 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.175+0000 7fe8c67fc640 10 monclient(hunting): my global_id is 4324 2024-03-25T18:18:40.175 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.175+0000 7fe8c67fc640 1 -- 172.21.15.97:0/3735760359 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8d0140e30 con 0x7fe8d014be90 2024-03-25T18:18:40.183 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.183+0000 7fe8c67fc640 1 -- 172.21.15.97:0/3735760359 <== mon.0 v1:172.21.15.97:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1570871529 0 0) 0x7fe8b4004eb0 con 0x7fe8d00b4d60 2024-03-25T18:18:40.183 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.183+0000 7fe8c67fc640 1 -- 172.21.15.97:0/3735760359 >> v1:172.21.15.97:6790/0 conn(0x7fe8d00b1730 legacy=0x7fe8d01d7690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:40.183 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.183+0000 7fe8c67fc640 1 -- 172.21.15.97:0/3735760359 >> v1:172.21.15.98:6789/0 conn(0x7fe8d014be90 legacy=0x7fe8d01430d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:40.184 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.183+0000 7fe8c67fc640 10 monclient: _finish_hunting 0 2024-03-25T18:18:40.184 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.183+0000 7fe8c67fc640 1 monclient: found mon.a 2024-03-25T18:18:40.184 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.183+0000 7fe8c67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:40.184 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.183+0000 7fe8c67fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:18:40.184 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.183+0000 7fe8c67fc640 1 -- 172.21.15.97:0/3735760359 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8d0140fc0 con 0x7fe8d00b4d60 2024-03-25T18:18:40.184 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.183+0000 7fe8c67fc640 10 monclient: _finish_auth 0 2024-03-25T18:18:40.184 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.183+0000 7fe8c67fc640 10 monclient: _check_auth_tickets 2024-03-25T18:18:40.184 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.183+0000 7fe8c67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:40.184 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.183+0000 7fe8d7d4b640 5 monclient: authenticate success, global_id 4326 2024-03-25T18:18:40.184 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.183+0000 7fe8d7d4b640 10 monclient: _renew_subs 2024-03-25T18:18:40.184 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.183+0000 7fe8d7d4b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:18:40.184 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.183+0000 7fe8d7d4b640 1 -- 172.21.15.97:0/3735760359 --> v1:172.21.15.97:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8d01385c0 con 0x7fe8d00b4d60 2024-03-25T18:18:40.184 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.183+0000 7fe8d7d4b640 10 client.4326.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:18:40.184 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.183+0000 7fe8d7d4b640 10 monclient: _renew_subs 2024-03-25T18:18:40.185 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.183+0000 7fe8d7d4b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:18:40.185 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.183+0000 7fe8d7d4b640 1 -- 172.21.15.97:0/3735760359 --> v1:172.21.15.97:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8d0137200 con 0x7fe8d00b4d60 2024-03-25T18:18:40.185 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.183+0000 7fe8d7d4b640 1 librados: init done 2024-03-25T18:18:40.185 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.184+0000 7fe8d7d4b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:40.185 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.184+0000 7fe8d7d4b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:40.185 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.184+0000 7fe8d7d4b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:18:40.185 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.184+0000 7fe8d7d4b640 1 -- 172.21.15.97:0/3735760359 --> v1:172.21.15.97:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8d0142a10 con 0x7fe8d00b4d60 2024-03-25T18:18:40.489 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.489+0000 7fe8c67fc640 1 -- 172.21.15.97:0/3735760359 <== mon.0 v1:172.21.15.97:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe8c0003610 con 0x7fe8d00b4d60 2024-03-25T18:18:40.489 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.489+0000 7fe8c67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:40.489 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.489+0000 7fe8c67fc640 1 -- 172.21.15.97:0/3735760359 <== mon.0 v1:172.21.15.97:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7fe8c0003a20 con 0x7fe8d00b4d60 2024-03-25T18:18:40.489 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.489+0000 7fe8c67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:40.489 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.489+0000 7fe8c67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:18:40.489 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.489+0000 7fe8c67fc640 10 monclient: dump: 2024-03-25T18:18:40.489 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:18:40.489 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:18:40.489 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:18:40.489 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:18:40.490 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:40.490 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:18:40.490 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:18:40.490 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:18:40.490 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:18:40.490 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:18:40.490 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.489+0000 7fe8c7fff640 4 set_mon_vals no callback set 2024-03-25T18:18:40.490 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.490+0000 7fe8c67fc640 1 -- 172.21.15.97:0/3735760359 <== mon.0 v1:172.21.15.97:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2377622137 0 0) 0x7fe8c0003c40 con 0x7fe8d00b4d60 2024-03-25T18:18:40.491 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.490+0000 7fe8d5ae7640 10 monclient: get_auth_request con 0x7fe8b406df30 auth_method 0 2024-03-25T18:18:40.491 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.490+0000 7fe8c67fc640 10 client.4326.objecter ms_handle_connect 0x7fe8b406df30 2024-03-25T18:18:40.497 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.496+0000 7fe8c67fc640 1 -- 172.21.15.97:0/3735760359 <== mon.0 v1:172.21.15.97:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 430711918 0 0) 0x7fe8c0054840 con 0x7fe8d00b4d60 2024-03-25T18:18:40.497 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.496+0000 7fe8c67fc640 10 client.4326.objecter ms_dispatch 0x7fe8d0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:18:40.497 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.496+0000 7fe8c67fc640 3 client.4326.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:18:40.497 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.496+0000 7fe8c67fc640 3 client.4326.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:18:40.497 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.496+0000 7fe8c67fc640 20 client.4326.objecter dump_active .. 0 homeless 2024-03-25T18:18:40.497 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.496+0000 7fe8c67fc640 1 -- 172.21.15.97:0/3735760359 <== mon.0 v1:172.21.15.97:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe8c0050350 con 0x7fe8d00b4d60 2024-03-25T18:18:40.497 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.497+0000 7fe8c67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:40.497 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.497+0000 7fe8c67fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:18:40.726 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.726+0000 7fe8d7d4b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:40.726 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.726+0000 7fe8d7d4b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:40.727 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.726+0000 7fe8d7d4b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:18:40.727 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.726+0000 7fe8d7d4b640 1 -- 172.21.15.97:0/3735760359 --> v1:172.21.15.97:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe8d0141150 con 0x7fe8d00b4d60 2024-03-25T18:18:40.761 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.761+0000 7fe8c67fc640 1 -- 172.21.15.97:0/3735760359 <== mon.0 v1:172.21.15.97:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2169427412) 0x7fe8d0141150 con 0x7fe8d00b4d60 2024-03-25T18:18:40.761 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.761+0000 7fe8c67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:40.761 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.761+0000 7fe8c67fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:18:40.762 INFO:teuthology.misc.health.smithi097.stdout: 2024-03-25T18:18:40.762 INFO:teuthology.misc.health.smithi097.stdout:{"epoch":10,"fsid":"4c3d472c-79b8-486b-9e3d-2738d4306754","created":"2024-03-25T18:16:14.737309+0000","modified":"2024-03-25T18:16:20.357492+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:20.357492+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cb5152b9-262f-45c3-94b8-18d0e81df558","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6daf8793-0838-473d-a7d6-9a0d2894c75d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"136f1493-3b9c-4f43-a2e7-641482d69dae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"397d538c-4f64-4daf-9e8b-662d09f91327","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4f1df08-663f-4bda-93f8-3878ab7c24b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c4b18d70-a3ce-4b15-9b70-eae5525bd761","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8dd4033a-de0e-4f8a-96dc-59f3c1f825b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0077722e-ffa6-4a3d-9891-929d9869b286","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:18:40.764 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.763+0000 7fe8d7d4b640 10 librados: watch_flush enter 2024-03-25T18:18:40.764 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.763+0000 7fe8d7d4b640 10 librados: watch_flush exit 2024-03-25T18:18:40.764 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.763+0000 7fe8d7d4b640 20 client.4326.objecter shutdown clearing up homeless session... 2024-03-25T18:18:40.765 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.763+0000 7fe8d7d4b640 10 client.4326.objecter successfully canceled tick 2024-03-25T18:18:40.765 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.764+0000 7fe8d7d4b640 1 -- 172.21.15.97:0/3735760359 >> v1:172.21.15.98:6800/800387573 conn(0x7fe8b406df30 legacy=0x7fe8b40703f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:40.765 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.764+0000 7fe8d7d4b640 10 monclient: shutdown 2024-03-25T18:18:40.765 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.764+0000 7fe8d7d4b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:40.765 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.764+0000 7fe8d7d4b640 1 -- 172.21.15.97:0/3735760359 >> v1:172.21.15.97:6789/0 conn(0x7fe8d00b4d60 legacy=0x7fe8d0140380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:40.765 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.764+0000 7fe8d7d4b640 1 -- 172.21.15.97:0/3735760359 shutdown_connections 2024-03-25T18:18:40.765 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.764+0000 7fe8d7d4b640 1 -- 172.21.15.97:0/3735760359 >> 172.21.15.97:0/3735760359 conn(0x7fe8d00617a0 msgr2=0x7fe8d013c3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:40.765 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.764+0000 7fe8d7d4b640 1 -- 172.21.15.97:0/3735760359 shutdown_connections 2024-03-25T18:18:40.765 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.764+0000 7fe8d7d4b640 1 -- 172.21.15.97:0/3735760359 wait complete. 2024-03-25T18:18:40.765 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:40.764+0000 7fe8d7d4b640 1 librados: shutdown 2024-03-25T18:18:40.776 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:18:47.583 DEBUG:teuthology.orchestra.run.smithi097:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:18:47.695 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:47.695+0000 7fc501193640 10 monclient: get_monmap_and_config 2024-03-25T18:18:47.695 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:47.695+0000 7fc501193640 10 monclient: build_initial_monmap 2024-03-25T18:18:47.696 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:47.695+0000 7fc501193640 10 monclient: monmap: 2024-03-25T18:18:47.696 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:18:47.696 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:18:47.696 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:18:47.695757+0000 2024-03-25T18:18:47.696 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:18:47.695757+0000 2024-03-25T18:18:47.696 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:47.696 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:18:47.696 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:18:47.696 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T18:18:47.696 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-b 2024-03-25T18:18:47.696 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:18:47.698 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:47.696+0000 7fc501193640 1 Processor -- start 2024-03-25T18:18:47.698 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:47.696+0000 7fc501193640 1 -- start start 2024-03-25T18:18:47.698 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:47.696+0000 7fc501193640 10 monclient: init 2024-03-25T18:18:47.698 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:47.697+0000 7fc501193640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:47.698 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:47.697+0000 7fc501193640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:18:47.698 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:47.697+0000 7fc501193640 10 monclient(hunting): picked mon.noname-c con 0x7fc4fc0a98f0 addr v1:172.21.15.98:6789/0 2024-03-25T18:18:47.698 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:47.697+0000 7fc501193640 10 monclient(hunting): picked mon.noname-b con 0x7fc4fc13f710 addr v1:172.21.15.97:6790/0 2024-03-25T18:18:47.698 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:47.697+0000 7fc501193640 10 monclient(hunting): picked mon.noname-a con 0x7fc4fc1430c0 addr v1:172.21.15.97:6789/0 2024-03-25T18:18:47.698 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:47.697+0000 7fc501193640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4fc142520 con 0x7fc4fc1430c0 2024-03-25T18:18:47.698 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:47.697+0000 7fc501193640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4fc0b28a0 con 0x7fc4fc0a98f0 2024-03-25T18:18:47.698 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:47.697+0000 7fc501193640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4fc0b2a30 con 0x7fc4fc13f710 2024-03-25T18:18:47.698 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:47.697+0000 7fc501193640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:47.698 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:47.697+0000 7fc501193640 10 monclient(hunting): authenticate will time out at 7739.972062s 2024-03-25T18:18:47.702 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:47.701+0000 7fc4fa59c640 1 --1- >> v1:172.21.15.97:6790/0 conn(0x7fc4fc13f710 0x7fc4fc1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6790/0 says I am v1:172.21.15.97:40040/0 (socket says 172.21.15.97:40040) 2024-03-25T18:18:47.702 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:47.701+0000 7fc4fa59c640 1 -- 172.21.15.97:0/152832923 learned_addr learned my addr 172.21.15.97:0/152832923 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:18:47.715 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:47.714+0000 7fc4f9d9b640 1 -- 172.21.15.97:0/152832923 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3277397462 0 0) 0x7fc4fc0b2a30 con 0x7fc4fc13f710 2024-03-25T18:18:47.715 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:47.714+0000 7fc4f9d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:18:47.715 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:47.714+0000 7fc4f9d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:47.715 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:47.714+0000 7fc4f9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:47.715 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:47.714+0000 7fc4f9d9b640 10 monclient(hunting): my global_id is 4322 2024-03-25T18:18:47.715 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:47.714+0000 7fc4f9d9b640 1 -- 172.21.15.97:0/152832923 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4e4003520 con 0x7fc4fc13f710 2024-03-25T18:18:47.726 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:47.726+0000 7fc4f9d9b640 1 -- 172.21.15.97:0/152832923 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 927863715 0 0) 0x7fc4fc0b28a0 con 0x7fc4fc0a98f0 2024-03-25T18:18:47.727 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:47.726+0000 7fc4f9d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:18:47.727 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:47.726+0000 7fc4f9d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:47.727 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:47.726+0000 7fc4f9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:47.727 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:47.726+0000 7fc4f9d9b640 10 monclient(hunting): my global_id is 4327 2024-03-25T18:18:47.727 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:47.726+0000 7fc4f9d9b640 1 -- 172.21.15.97:0/152832923 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4fc0b2a30 con 0x7fc4fc0a98f0 2024-03-25T18:18:47.729 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:47.729+0000 7fc4f9d9b640 1 -- 172.21.15.97:0/152832923 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4143368134 0 0) 0x7fc4fc142520 con 0x7fc4fc1430c0 2024-03-25T18:18:47.730 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:47.729+0000 7fc4f9d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:18:47.730 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:47.729+0000 7fc4f9d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:47.730 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:47.729+0000 7fc4f9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:47.730 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:47.729+0000 7fc4f9d9b640 10 monclient(hunting): my global_id is 4329 2024-03-25T18:18:47.730 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:47.729+0000 7fc4f9d9b640 1 -- 172.21.15.97:0/152832923 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4fc0b28a0 con 0x7fc4fc1430c0 2024-03-25T18:18:47.734 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:47.734+0000 7fc4f9d9b640 1 -- 172.21.15.97:0/152832923 <== mon.2 v1:172.21.15.97:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 5859025 0 0) 0x7fc4e4003520 con 0x7fc4fc13f710 2024-03-25T18:18:47.735 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:47.734+0000 7fc4f9d9b640 1 -- 172.21.15.97:0/152832923 >> v1:172.21.15.98:6789/0 conn(0x7fc4fc0a98f0 legacy=0x7fc4fc0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:47.735 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:47.734+0000 7fc4f9d9b640 1 -- 172.21.15.97:0/152832923 >> v1:172.21.15.97:6789/0 conn(0x7fc4fc1430c0 legacy=0x7fc4fc14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:47.735 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:47.734+0000 7fc4f9d9b640 10 monclient: _finish_hunting 0 2024-03-25T18:18:47.735 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:47.734+0000 7fc4f9d9b640 1 monclient: found mon.noname-b 2024-03-25T18:18:47.735 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:47.734+0000 7fc4f9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:47.735 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:47.734+0000 7fc4f9d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.97:6790/0 2024-03-25T18:18:47.735 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:47.734+0000 7fc4f9d9b640 1 -- 172.21.15.97:0/152832923 --> v1:172.21.15.97:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4fc0b44c0 con 0x7fc4fc13f710 2024-03-25T18:18:47.735 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:47.734+0000 7fc4f9d9b640 10 monclient: _finish_auth 0 2024-03-25T18:18:47.735 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:47.734+0000 7fc4f9d9b640 10 monclient: _check_auth_tickets 2024-03-25T18:18:47.735 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:47.734+0000 7fc4f9d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:47.735 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:47.734+0000 7fc4f9d9b640 1 -- 172.21.15.97:0/152832923 <== mon.2 v1:172.21.15.97:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7fc4f0001690 con 0x7fc4fc13f710 2024-03-25T18:18:47.735 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:47.734+0000 7fc501193640 5 monclient: authenticate success, global_id 4322 2024-03-25T18:18:47.735 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:47.734+0000 7fc501193640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:18:47.735 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:47.734+0000 7fc4f9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:47.736 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:47.734+0000 7fc4f9d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:18:47.736 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:47.734+0000 7fc4f9d9b640 10 monclient: dump: 2024-03-25T18:18:47.736 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:18:47.736 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:18:47.736 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:18:47.736 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:18:47.736 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:47.736 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:18:47.736 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:18:47.736 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:18:47.736 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:18:47.736 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:18:47.736 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:47.734+0000 7fc501193640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:18:47.750 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:47.750+0000 7fc4f9d9b640 1 -- 172.21.15.97:0/152832923 <== mon.2 v1:172.21.15.97:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc4f00017f0 con 0x7fc4fc13f710 2024-03-25T18:18:47.750 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:47.750+0000 7fc4f9d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:47.750 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:47.750+0000 7fc4f9d9b640 1 -- 172.21.15.97:0/152832923 <== mon.2 v1:172.21.15.97:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7fc4f0001c00 con 0x7fc4fc13f710 2024-03-25T18:18:47.750 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:47.750+0000 7fc4f9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:47.750 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:47.750+0000 7fc4f9d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:18:47.751 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:47.750+0000 7fc4f9d9b640 10 monclient: dump: 2024-03-25T18:18:47.751 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:18:47.751 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:18:47.751 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:18:47.751 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:18:47.751 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:47.751 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:18:47.751 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:18:47.751 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:18:47.751 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:18:47.751 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:18:47.751 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:47.750+0000 7fc501193640 10 monclient: get_monmap_and_config success 2024-03-25T18:18:47.752 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:47.750+0000 7fc501193640 4 set_mon_vals no callback set 2024-03-25T18:18:47.753 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:47.751+0000 7fc501193640 10 monclient: shutdown 2024-03-25T18:18:47.754 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:47.751+0000 7fc501193640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:47.754 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:47.751+0000 7fc501193640 1 -- 172.21.15.97:0/152832923 >> v1:172.21.15.97:6790/0 conn(0x7fc4fc13f710 legacy=0x7fc4fc1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:47.754 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:47.751+0000 7fc501193640 1 -- 172.21.15.97:0/152832923 shutdown_connections 2024-03-25T18:18:47.754 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:47.751+0000 7fc501193640 1 -- 172.21.15.97:0/152832923 >> 172.21.15.97:0/152832923 conn(0x7fc4fc0b7ea0 msgr2=0x7fc4fc0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:47.754 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:47.751+0000 7fc501193640 1 -- 172.21.15.97:0/152832923 shutdown_connections 2024-03-25T18:18:47.754 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:47.751+0000 7fc501193640 1 -- 172.21.15.97:0/152832923 wait complete. 2024-03-25T18:18:47.754 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:47.752+0000 7fc501193640 10 monclient: build_initial_monmap 2024-03-25T18:18:47.754 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:47.752+0000 7fc501193640 10 monclient: monmap: 2024-03-25T18:18:47.754 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:18:47.754 INFO:teuthology.misc.health.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:18:47.754 INFO:teuthology.misc.health.smithi097.stderr:last_changed 0.000000 2024-03-25T18:18:47.755 INFO:teuthology.misc.health.smithi097.stderr:created 0.000000 2024-03-25T18:18:47.755 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:47.755 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 1 2024-03-25T18:18:47.755 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:18:47.755 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T18:18:47.755 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-c 2024-03-25T18:18:47.755 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:18:47.755 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:47.752+0000 7fc501193640 1 librados: starting msgr at 2024-03-25T18:18:47.755 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:47.752+0000 7fc501193640 1 librados: starting objecter 2024-03-25T18:18:47.755 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:47.752+0000 7fc501193640 1 Processor -- start 2024-03-25T18:18:47.755 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:47.752+0000 7fc501193640 1 -- start start 2024-03-25T18:18:47.755 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:47.752+0000 7fc501193640 1 librados: setting wanted keys 2024-03-25T18:18:47.755 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:47.752+0000 7fc501193640 1 librados: calling monclient init 2024-03-25T18:18:47.755 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:47.752+0000 7fc501193640 10 monclient: init 2024-03-25T18:18:47.756 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:47.753+0000 7fc501193640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:47.756 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:47.753+0000 7fc501193640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:18:47.756 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:47.753+0000 7fc501193640 10 monclient(hunting): picked mon.noname-a con 0x7fc4fc0a98f0 addr v1:172.21.15.97:6789/0 2024-03-25T18:18:47.756 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:47.753+0000 7fc501193640 10 monclient(hunting): picked mon.noname-b con 0x7fc4fc13f710 addr v1:172.21.15.98:6789/0 2024-03-25T18:18:47.756 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:47.753+0000 7fc501193640 10 monclient(hunting): picked mon.noname-c con 0x7fc4fc1430c0 addr v1:172.21.15.97:6790/0 2024-03-25T18:18:47.756 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:47.753+0000 7fc501193640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4fc1370a0 con 0x7fc4fc0a98f0 2024-03-25T18:18:47.756 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:47.753+0000 7fc501193640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4fc135090 con 0x7fc4fc13f710 2024-03-25T18:18:47.756 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:47.753+0000 7fc501193640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4fc135220 con 0x7fc4fc1430c0 2024-03-25T18:18:47.756 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:47.753+0000 7fc501193640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:47.756 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:47.753+0000 7fc501193640 10 monclient(hunting): authenticate will time out at 7740.028848s 2024-03-25T18:18:47.763 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:47.763+0000 7fc4fad9d640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fc4fc0a98f0 0x7fc4fc1dae50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:55492/0 (socket says 172.21.15.97:55492) 2024-03-25T18:18:47.763 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:47.763+0000 7fc4fad9d640 1 -- 172.21.15.97:0/3251747790 learned_addr learned my addr 172.21.15.97:0/3251747790 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:18:47.765 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:47.765+0000 7fc4e37fe640 10 client.?.objecter ms_handle_connect 0x7fc4fc0a98f0 2024-03-25T18:18:47.765 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:47.765+0000 7fc4e37fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:47.767 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:47.767+0000 7fc4e37fe640 10 client.?.objecter ms_handle_connect 0x7fc4fc13f710 2024-03-25T18:18:47.767 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:47.767+0000 7fc4e37fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:47.770 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:47.770+0000 7fc4e37fe640 1 -- 172.21.15.97:0/3251747790 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2100389913 0 0) 0x7fc4fc1370a0 con 0x7fc4fc0a98f0 2024-03-25T18:18:47.770 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:47.770+0000 7fc4e37fe640 10 monclient(hunting): _negotiate 2024-03-25T18:18:47.770 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:47.770+0000 7fc4e37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:47.771 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:47.770+0000 7fc4e37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:47.771 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:47.770+0000 7fc4e37fe640 10 monclient(hunting): my global_id is 4332 2024-03-25T18:18:47.771 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:47.770+0000 7fc4e37fe640 1 -- 172.21.15.97:0/3251747790 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4e8004eb0 con 0x7fc4fc0a98f0 2024-03-25T18:18:47.782 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:47.782+0000 7fc4e37fe640 1 -- 172.21.15.97:0/3251747790 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2730991806 0 0) 0x7fc4fc135090 con 0x7fc4fc13f710 2024-03-25T18:18:47.782 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:47.782+0000 7fc4e37fe640 10 monclient(hunting): _negotiate 2024-03-25T18:18:47.782 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:47.782+0000 7fc4e37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:47.782 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:47.782+0000 7fc4e37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:47.782 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:47.782+0000 7fc4e37fe640 10 monclient(hunting): my global_id is 4330 2024-03-25T18:18:47.782 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:47.782+0000 7fc4e37fe640 1 -- 172.21.15.97:0/3251747790 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4fc1370a0 con 0x7fc4fc13f710 2024-03-25T18:18:47.782 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:47.782+0000 7fc4e37fe640 1 -- 172.21.15.97:0/3251747790 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 318741905 0 0) 0x7fc4e8004eb0 con 0x7fc4fc0a98f0 2024-03-25T18:18:47.783 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:47.782+0000 7fc4e37fe640 1 -- 172.21.15.97:0/3251747790 --> v1:172.21.15.97:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc4fc135090 con 0x7fc4fc0a98f0 2024-03-25T18:18:47.783 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:47.782+0000 7fc4e37fe640 1 -- 172.21.15.97:0/3251747790 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7fc4ec001a30 con 0x7fc4fc0a98f0 2024-03-25T18:18:47.783 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:47.782+0000 7fc4e37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:47.783 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:47.782+0000 7fc4e37fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:18:47.783 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:47.782+0000 7fc4e37fe640 10 monclient(hunting): dump: 2024-03-25T18:18:47.783 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:18:47.783 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:18:47.783 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:18:47.783 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:18:47.783 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:47.783 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:18:47.783 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:18:47.783 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:18:47.783 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:18:47.783 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:18:47.784 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:47.782+0000 7fc4e37fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:18:47.786 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:47.786+0000 7fc4e37fe640 10 client.?.objecter ms_handle_connect 0x7fc4fc1430c0 2024-03-25T18:18:47.786 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:47.786+0000 7fc4e37fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:47.791 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:47.791+0000 7fc4e37fe640 1 -- 172.21.15.97:0/3251747790 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2474031574 0 0) 0x7fc4fc1370a0 con 0x7fc4fc13f710 2024-03-25T18:18:47.791 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:47.791+0000 7fc4e37fe640 1 -- 172.21.15.97:0/3251747790 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc4e8004eb0 con 0x7fc4fc13f710 2024-03-25T18:18:47.791 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:47.791+0000 7fc4e37fe640 1 -- 172.21.15.97:0/3251747790 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7fc4f0001b00 con 0x7fc4fc13f710 2024-03-25T18:18:47.791 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:47.791+0000 7fc4e37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:47.791 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:47.791+0000 7fc4e37fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:18:47.791 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:47.791+0000 7fc4e37fe640 10 monclient(hunting): dump: 2024-03-25T18:18:47.791 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:18:47.792 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:18:47.792 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:18:47.792 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:18:47.792 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:47.792 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:18:47.792 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:18:47.792 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:18:47.792 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:18:47.792 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:18:47.798 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:47.798+0000 7fc4e37fe640 1 -- 172.21.15.97:0/3251747790 <== mon.0 v1:172.21.15.97:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2440504248 0 0) 0x7fc4fc135090 con 0x7fc4fc0a98f0 2024-03-25T18:18:47.798 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:47.798+0000 7fc4e37fe640 1 -- 172.21.15.97:0/3251747790 >> v1:172.21.15.97:6790/0 conn(0x7fc4fc1430c0 legacy=0x7fc4fc1d48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:47.798 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:47.798+0000 7fc4e37fe640 1 -- 172.21.15.97:0/3251747790 >> v1:172.21.15.98:6789/0 conn(0x7fc4fc13f710 legacy=0x7fc4fc1d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:47.798 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:47.798+0000 7fc4e37fe640 10 monclient: _finish_hunting 0 2024-03-25T18:18:47.799 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:47.798+0000 7fc4e37fe640 1 monclient: found mon.a 2024-03-25T18:18:47.799 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:47.798+0000 7fc4e37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:47.799 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:47.798+0000 7fc4e37fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:18:47.799 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:47.798+0000 7fc4e37fe640 1 -- 172.21.15.97:0/3251747790 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4fc1d5000 con 0x7fc4fc0a98f0 2024-03-25T18:18:47.799 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:47.798+0000 7fc4e37fe640 10 monclient: _finish_auth 0 2024-03-25T18:18:47.799 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:47.798+0000 7fc4e37fe640 10 monclient: _check_auth_tickets 2024-03-25T18:18:47.799 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:47.798+0000 7fc4e37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:47.799 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:47.798+0000 7fc501193640 5 monclient: authenticate success, global_id 4332 2024-03-25T18:18:47.799 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:47.798+0000 7fc501193640 10 monclient: _renew_subs 2024-03-25T18:18:47.799 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:47.798+0000 7fc501193640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:18:47.799 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:47.798+0000 7fc501193640 1 -- 172.21.15.97:0/3251747790 --> v1:172.21.15.97:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4fc1d52b0 con 0x7fc4fc0a98f0 2024-03-25T18:18:47.799 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:47.798+0000 7fc501193640 10 client.4332.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:18:47.799 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:47.798+0000 7fc501193640 10 monclient: _renew_subs 2024-03-25T18:18:47.799 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:47.798+0000 7fc501193640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:18:47.799 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:47.798+0000 7fc501193640 1 -- 172.21.15.97:0/3251747790 --> v1:172.21.15.97:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4fc135910 con 0x7fc4fc0a98f0 2024-03-25T18:18:47.800 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:47.798+0000 7fc501193640 1 librados: init done 2024-03-25T18:18:47.800 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:47.799+0000 7fc501193640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:47.800 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:47.799+0000 7fc501193640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:47.800 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:47.799+0000 7fc501193640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:18:47.800 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:47.799+0000 7fc501193640 1 -- 172.21.15.97:0/3251747790 --> v1:172.21.15.97:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4fc0627b0 con 0x7fc4fc0a98f0 2024-03-25T18:18:48.126 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:48.126+0000 7fc4e37fe640 1 -- 172.21.15.97:0/3251747790 <== mon.0 v1:172.21.15.97:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc4ec001c90 con 0x7fc4fc0a98f0 2024-03-25T18:18:48.127 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:48.126+0000 7fc4e37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:48.127 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:48.126+0000 7fc4e37fe640 1 -- 172.21.15.97:0/3251747790 <== mon.0 v1:172.21.15.97:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7fc4ec003a60 con 0x7fc4fc0a98f0 2024-03-25T18:18:48.127 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:48.126+0000 7fc4e37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:48.127 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:48.126+0000 7fc4e37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:18:48.127 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:48.126+0000 7fc4e37fe640 10 monclient: dump: 2024-03-25T18:18:48.127 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:18:48.127 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:18:48.127 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:18:48.127 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:18:48.127 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:48.127 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:18:48.127 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:18:48.127 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:18:48.127 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:18:48.128 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:18:48.128 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:48.126+0000 7fc4f959a640 4 set_mon_vals no callback set 2024-03-25T18:18:48.128 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:48.127+0000 7fc4e37fe640 1 -- 172.21.15.97:0/3251747790 <== mon.0 v1:172.21.15.97:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2377622137 0 0) 0x7fc4ec003c80 con 0x7fc4fc0a98f0 2024-03-25T18:18:48.128 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:48.127+0000 7fc4fa59c640 10 monclient: get_auth_request con 0x7fc4e806de40 auth_method 0 2024-03-25T18:18:48.128 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:48.128+0000 7fc4e37fe640 10 client.4332.objecter ms_handle_connect 0x7fc4e806de40 2024-03-25T18:18:48.133 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:48.133+0000 7fc4e37fe640 1 -- 172.21.15.97:0/3251747790 <== mon.0 v1:172.21.15.97:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 430711918 0 0) 0x7fc4ec054880 con 0x7fc4fc0a98f0 2024-03-25T18:18:48.134 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:48.133+0000 7fc4e37fe640 10 client.4332.objecter ms_dispatch 0x7fc4fc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:18:48.134 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:48.133+0000 7fc4e37fe640 3 client.4332.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:18:48.134 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:48.133+0000 7fc4e37fe640 3 client.4332.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:18:48.134 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:48.133+0000 7fc4e37fe640 20 client.4332.objecter dump_active .. 0 homeless 2024-03-25T18:18:48.134 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:48.133+0000 7fc4e37fe640 1 -- 172.21.15.97:0/3251747790 <== mon.0 v1:172.21.15.97:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc4ec050390 con 0x7fc4fc0a98f0 2024-03-25T18:18:48.134 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:48.133+0000 7fc4e37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:48.134 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:48.133+0000 7fc4e37fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:18:48.347 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:48.347+0000 7fc501193640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:48.347 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:48.347+0000 7fc501193640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:48.347 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:48.347+0000 7fc501193640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:18:48.348 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:48.347+0000 7fc501193640 1 -- 172.21.15.97:0/3251747790 --> v1:172.21.15.97:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc4fc13bfe0 con 0x7fc4fc0a98f0 2024-03-25T18:18:48.370 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:48.370+0000 7fc4e37fe640 1 -- 172.21.15.97:0/3251747790 <== mon.0 v1:172.21.15.97:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2169427412) 0x7fc4ec0522c0 con 0x7fc4fc0a98f0 2024-03-25T18:18:48.370 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:48.370+0000 7fc4e37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:48.370 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:48.370+0000 7fc4e37fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:18:48.370 INFO:teuthology.misc.health.smithi097.stdout: 2024-03-25T18:18:48.371 INFO:teuthology.misc.health.smithi097.stdout:{"epoch":10,"fsid":"4c3d472c-79b8-486b-9e3d-2738d4306754","created":"2024-03-25T18:16:14.737309+0000","modified":"2024-03-25T18:16:20.357492+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:20.357492+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cb5152b9-262f-45c3-94b8-18d0e81df558","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6daf8793-0838-473d-a7d6-9a0d2894c75d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"136f1493-3b9c-4f43-a2e7-641482d69dae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"397d538c-4f64-4daf-9e8b-662d09f91327","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4f1df08-663f-4bda-93f8-3878ab7c24b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c4b18d70-a3ce-4b15-9b70-eae5525bd761","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8dd4033a-de0e-4f8a-96dc-59f3c1f825b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0077722e-ffa6-4a3d-9891-929d9869b286","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:18:48.373 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:48.373+0000 7fc501193640 10 librados: watch_flush enter 2024-03-25T18:18:48.373 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:48.373+0000 7fc501193640 10 librados: watch_flush exit 2024-03-25T18:18:48.373 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:48.373+0000 7fc501193640 20 client.4332.objecter shutdown clearing up homeless session... 2024-03-25T18:18:48.373 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:48.373+0000 7fc501193640 10 client.4332.objecter successfully canceled tick 2024-03-25T18:18:48.374 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:48.373+0000 7fc501193640 1 -- 172.21.15.97:0/3251747790 >> v1:172.21.15.98:6800/800387573 conn(0x7fc4e806de40 legacy=0x7fc4e8070300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:48.374 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:48.373+0000 7fc501193640 10 monclient: shutdown 2024-03-25T18:18:48.374 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:48.373+0000 7fc501193640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:48.374 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:48.373+0000 7fc501193640 1 -- 172.21.15.97:0/3251747790 >> v1:172.21.15.97:6789/0 conn(0x7fc4fc0a98f0 legacy=0x7fc4fc1dae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:48.374 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:48.373+0000 7fc501193640 1 -- 172.21.15.97:0/3251747790 shutdown_connections 2024-03-25T18:18:48.374 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:48.373+0000 7fc501193640 1 -- 172.21.15.97:0/3251747790 >> 172.21.15.97:0/3251747790 conn(0x7fc4fc0b7ea0 msgr2=0x7fc4fc074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:48.374 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:48.373+0000 7fc501193640 1 -- 172.21.15.97:0/3251747790 shutdown_connections 2024-03-25T18:18:48.374 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:48.373+0000 7fc501193640 1 -- 172.21.15.97:0/3251747790 wait complete. 2024-03-25T18:18:48.374 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:48.373+0000 7fc501193640 1 librados: shutdown 2024-03-25T18:18:48.385 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:18:52.175 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:02:54.214 34931== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:52.217 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:02:54.260 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:52.219 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:02:54.262 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:52.224 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:02:54.263 34931== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:52.265 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:02:54.308 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:52.267 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:02:54.306 34931== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:52.280 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:02:54.319 34935== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:52.298 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:02:54.337 34935== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:52.321 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:02:54.360 34935== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:55.191 DEBUG:teuthology.orchestra.run.smithi097:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:18:55.308 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.306+0000 7f5d3a486640 10 monclient: get_monmap_and_config 2024-03-25T18:18:55.308 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.306+0000 7f5d3a486640 10 monclient: build_initial_monmap 2024-03-25T18:18:55.308 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.307+0000 7f5d3a486640 10 monclient: monmap: 2024-03-25T18:18:55.308 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:18:55.308 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:18:55.308 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:18:55.308251+0000 2024-03-25T18:18:55.308 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:18:55.308251+0000 2024-03-25T18:18:55.308 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:55.308 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:18:55.309 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:18:55.309 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T18:18:55.309 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-b 2024-03-25T18:18:55.309 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:18:55.310 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.307+0000 7f5d3a486640 1 Processor -- start 2024-03-25T18:18:55.310 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.307+0000 7f5d3a486640 1 -- start start 2024-03-25T18:18:55.310 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.307+0000 7f5d3a486640 10 monclient: init 2024-03-25T18:18:55.310 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.308+0000 7f5d3a486640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:55.310 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.308+0000 7f5d3a486640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:18:55.310 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.308+0000 7f5d3a486640 10 monclient(hunting): picked mon.noname-b con 0x7f5d340a98f0 addr v1:172.21.15.97:6790/0 2024-03-25T18:18:55.310 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.308+0000 7f5d3a486640 10 monclient(hunting): picked mon.noname-a con 0x7f5d3413f710 addr v1:172.21.15.97:6789/0 2024-03-25T18:18:55.310 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.308+0000 7f5d3a486640 10 monclient(hunting): picked mon.noname-c con 0x7f5d341430c0 addr v1:172.21.15.98:6789/0 2024-03-25T18:18:55.310 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.308+0000 7f5d3a486640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d34142520 con 0x7f5d3413f710 2024-03-25T18:18:55.311 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.308+0000 7f5d3a486640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d340b28a0 con 0x7f5d341430c0 2024-03-25T18:18:55.311 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.308+0000 7f5d3a486640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d340b2a30 con 0x7f5d340a98f0 2024-03-25T18:18:55.311 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.308+0000 7f5d3a486640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:55.311 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.308+0000 7f5d3a486640 10 monclient(hunting): authenticate will time out at 7747.584721s 2024-03-25T18:18:55.326 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.324+0000 7f5d33fff640 1 --1- >> v1:172.21.15.97:6790/0 conn(0x7f5d340a98f0 0x7f5d340a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6790/0 says I am v1:172.21.15.97:54820/0 (socket says 172.21.15.97:54820) 2024-03-25T18:18:55.326 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.324+0000 7f5d33fff640 1 -- 172.21.15.97:0/4071719758 learned_addr learned my addr 172.21.15.97:0/4071719758 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:18:55.335 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.334+0000 7f5d32ffd640 1 -- 172.21.15.97:0/4071719758 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 315590343 0 0) 0x7f5d340b28a0 con 0x7f5d341430c0 2024-03-25T18:18:55.335 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.334+0000 7f5d32ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:18:55.335 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.334+0000 7f5d32ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:55.335 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.334+0000 7f5d32ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:55.335 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.334+0000 7f5d32ffd640 10 monclient(hunting): my global_id is 4333 2024-03-25T18:18:55.335 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.334+0000 7f5d32ffd640 1 -- 172.21.15.97:0/4071719758 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d1c003520 con 0x7f5d341430c0 2024-03-25T18:18:55.337 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.336+0000 7f5d32ffd640 1 -- 172.21.15.97:0/4071719758 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1132262066 0 0) 0x7f5d34142520 con 0x7f5d3413f710 2024-03-25T18:18:55.337 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.336+0000 7f5d32ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:18:55.337 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.336+0000 7f5d32ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:55.338 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.336+0000 7f5d32ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:55.338 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.336+0000 7f5d32ffd640 10 monclient(hunting): my global_id is 4335 2024-03-25T18:18:55.338 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.336+0000 7f5d32ffd640 1 -- 172.21.15.97:0/4071719758 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d340b28a0 con 0x7f5d3413f710 2024-03-25T18:18:55.342 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.341+0000 7f5d32ffd640 1 -- 172.21.15.97:0/4071719758 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 346195655 0 0) 0x7f5d340b2a30 con 0x7f5d340a98f0 2024-03-25T18:18:55.342 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.341+0000 7f5d32ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:18:55.342 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.341+0000 7f5d32ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:55.342 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.341+0000 7f5d32ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:55.342 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.341+0000 7f5d32ffd640 10 monclient(hunting): my global_id is 4328 2024-03-25T18:18:55.342 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.341+0000 7f5d32ffd640 1 -- 172.21.15.97:0/4071719758 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d34142520 con 0x7f5d340a98f0 2024-03-25T18:18:55.345 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.343+0000 7f5d32ffd640 1 -- 172.21.15.97:0/4071719758 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1669527716 0 0) 0x7f5d1c003520 con 0x7f5d341430c0 2024-03-25T18:18:55.345 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.343+0000 7f5d32ffd640 1 -- 172.21.15.97:0/4071719758 >> v1:172.21.15.97:6790/0 conn(0x7f5d340a98f0 legacy=0x7f5d340a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:55.345 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.343+0000 7f5d32ffd640 1 -- 172.21.15.97:0/4071719758 >> v1:172.21.15.97:6789/0 conn(0x7f5d3413f710 legacy=0x7f5d341429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:55.345 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.343+0000 7f5d32ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:18:55.345 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.343+0000 7f5d32ffd640 1 monclient: found mon.noname-c 2024-03-25T18:18:55.345 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.343+0000 7f5d32ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:55.345 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.343+0000 7f5d32ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.98:6789/0 2024-03-25T18:18:55.345 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.343+0000 7f5d32ffd640 1 -- 172.21.15.97:0/4071719758 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d340b44c0 con 0x7f5d341430c0 2024-03-25T18:18:55.345 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.343+0000 7f5d32ffd640 10 monclient: _finish_auth 0 2024-03-25T18:18:55.345 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.343+0000 7f5d32ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:18:55.345 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.343+0000 7f5d32ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:55.346 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.343+0000 7f5d32ffd640 1 -- 172.21.15.97:0/4071719758 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f5d2c001a10 con 0x7f5d341430c0 2024-03-25T18:18:55.346 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.343+0000 7f5d32ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:55.346 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.343+0000 7f5d32ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:18:55.346 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.343+0000 7f5d32ffd640 10 monclient: dump: 2024-03-25T18:18:55.346 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:18:55.346 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:18:55.346 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:18:55.346 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:18:55.346 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:55.346 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:18:55.346 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:18:55.346 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:18:55.346 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:18:55.346 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:18:55.347 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.343+0000 7f5d3a486640 5 monclient: authenticate success, global_id 4333 2024-03-25T18:18:55.347 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.343+0000 7f5d3a486640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:18:55.362 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.361+0000 7f5d32ffd640 1 -- 172.21.15.97:0/4071719758 <== mon.1 v1:172.21.15.98:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5d2c001b70 con 0x7f5d341430c0 2024-03-25T18:18:55.363 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.361+0000 7f5d32ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:55.363 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.361+0000 7f5d32ffd640 1 -- 172.21.15.97:0/4071719758 <== mon.1 v1:172.21.15.98:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f5d2c001f80 con 0x7f5d341430c0 2024-03-25T18:18:55.363 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.361+0000 7f5d3a486640 10 monclient: get_monmap_and_config success 2024-03-25T18:18:55.363 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.361+0000 7f5d3a486640 4 set_mon_vals no callback set 2024-03-25T18:18:55.363 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.361+0000 7f5d32ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:55.363 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.361+0000 7f5d32ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:18:55.363 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.361+0000 7f5d32ffd640 10 monclient: dump: 2024-03-25T18:18:55.363 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:18:55.363 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:18:55.363 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:18:55.363 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:18:55.363 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:55.363 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:18:55.363 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:18:55.363 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:18:55.364 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:18:55.364 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:18:55.365 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.362+0000 7f5d3a486640 10 monclient: shutdown 2024-03-25T18:18:55.365 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.362+0000 7f5d3a486640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:55.365 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.362+0000 7f5d3a486640 1 -- 172.21.15.97:0/4071719758 >> v1:172.21.15.98:6789/0 conn(0x7f5d341430c0 legacy=0x7f5d3414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:55.365 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.362+0000 7f5d3a486640 1 -- 172.21.15.97:0/4071719758 shutdown_connections 2024-03-25T18:18:55.365 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.362+0000 7f5d3a486640 1 -- 172.21.15.97:0/4071719758 >> 172.21.15.97:0/4071719758 conn(0x7f5d340b7ea0 msgr2=0x7f5d340b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:55.365 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.362+0000 7f5d3a486640 1 -- 172.21.15.97:0/4071719758 shutdown_connections 2024-03-25T18:18:55.365 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.362+0000 7f5d3a486640 1 -- 172.21.15.97:0/4071719758 wait complete. 2024-03-25T18:18:55.365 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.363+0000 7f5d3a486640 10 monclient: build_initial_monmap 2024-03-25T18:18:55.365 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.363+0000 7f5d3a486640 10 monclient: monmap: 2024-03-25T18:18:55.366 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:18:55.366 INFO:teuthology.misc.health.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:18:55.366 INFO:teuthology.misc.health.smithi097.stderr:last_changed 0.000000 2024-03-25T18:18:55.366 INFO:teuthology.misc.health.smithi097.stderr:created 0.000000 2024-03-25T18:18:55.366 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:55.366 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 1 2024-03-25T18:18:55.366 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:18:55.366 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T18:18:55.366 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-c 2024-03-25T18:18:55.366 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:18:55.366 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.363+0000 7f5d3a486640 1 librados: starting msgr at 2024-03-25T18:18:55.366 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.363+0000 7f5d3a486640 1 librados: starting objecter 2024-03-25T18:18:55.366 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.363+0000 7f5d3a486640 1 Processor -- start 2024-03-25T18:18:55.366 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.363+0000 7f5d3a486640 1 -- start start 2024-03-25T18:18:55.366 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.363+0000 7f5d3a486640 1 librados: setting wanted keys 2024-03-25T18:18:55.367 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.363+0000 7f5d3a486640 1 librados: calling monclient init 2024-03-25T18:18:55.367 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.363+0000 7f5d3a486640 10 monclient: init 2024-03-25T18:18:55.367 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.364+0000 7f5d3a486640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:55.368 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.364+0000 7f5d3a486640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:18:55.368 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.364+0000 7f5d3a486640 10 monclient(hunting): picked mon.noname-a con 0x7f5d340a98f0 addr v1:172.21.15.97:6789/0 2024-03-25T18:18:55.368 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.364+0000 7f5d3a486640 10 monclient(hunting): picked mon.noname-b con 0x7f5d3413f710 addr v1:172.21.15.98:6789/0 2024-03-25T18:18:55.368 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.364+0000 7f5d3a486640 10 monclient(hunting): picked mon.noname-c con 0x7f5d341430c0 addr v1:172.21.15.97:6790/0 2024-03-25T18:18:55.368 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.364+0000 7f5d3a486640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d340ab920 con 0x7f5d340a98f0 2024-03-25T18:18:55.368 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.364+0000 7f5d3a486640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d34135140 con 0x7f5d3413f710 2024-03-25T18:18:55.368 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.364+0000 7f5d3a486640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d34134e10 con 0x7f5d341430c0 2024-03-25T18:18:55.368 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.364+0000 7f5d3a486640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:55.368 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.364+0000 7f5d3a486640 10 monclient(hunting): authenticate will time out at 7747.640728s 2024-03-25T18:18:55.370 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.369+0000 7f5d337fe640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f5d3413f710 0x7f5d34133ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.97:36372/0 (socket says 172.21.15.97:36372) 2024-03-25T18:18:55.370 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.369+0000 7f5d337fe640 1 -- 172.21.15.97:0/2136096934 learned_addr learned my addr 172.21.15.97:0/2136096934 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:18:55.375 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.374+0000 7f5d30ff9640 10 client.?.objecter ms_handle_connect 0x7f5d341430c0 2024-03-25T18:18:55.376 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.374+0000 7f5d30ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:55.377 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.376+0000 7f5d30ff9640 10 client.?.objecter ms_handle_connect 0x7f5d3413f710 2024-03-25T18:18:55.377 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.376+0000 7f5d30ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:55.384 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.383+0000 7f5d30ff9640 10 client.?.objecter ms_handle_connect 0x7f5d340a98f0 2024-03-25T18:18:55.384 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.383+0000 7f5d30ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:55.386 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.385+0000 7f5d30ff9640 1 -- 172.21.15.97:0/2136096934 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1898720113 0 0) 0x7f5d34135140 con 0x7f5d3413f710 2024-03-25T18:18:55.386 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.385+0000 7f5d30ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:18:55.386 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.385+0000 7f5d30ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:55.386 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.385+0000 7f5d30ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:55.386 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.385+0000 7f5d30ff9640 10 monclient(hunting): my global_id is 4336 2024-03-25T18:18:55.387 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.385+0000 7f5d30ff9640 1 -- 172.21.15.97:0/2136096934 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d20004eb0 con 0x7f5d3413f710 2024-03-25T18:18:55.388 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.386+0000 7f5d30ff9640 1 -- 172.21.15.97:0/2136096934 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2614563372 0 0) 0x7f5d34134e10 con 0x7f5d341430c0 2024-03-25T18:18:55.388 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.386+0000 7f5d30ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:18:55.388 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.386+0000 7f5d30ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:55.388 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.386+0000 7f5d30ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:55.388 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.386+0000 7f5d30ff9640 10 monclient(hunting): my global_id is 4331 2024-03-25T18:18:55.388 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.386+0000 7f5d30ff9640 1 -- 172.21.15.97:0/2136096934 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d34135140 con 0x7f5d341430c0 2024-03-25T18:18:55.391 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.390+0000 7f5d30ff9640 1 -- 172.21.15.97:0/2136096934 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1897991672 0 0) 0x7f5d340ab920 con 0x7f5d340a98f0 2024-03-25T18:18:55.391 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.390+0000 7f5d30ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:18:55.391 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.390+0000 7f5d30ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:55.391 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.390+0000 7f5d30ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:55.391 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.390+0000 7f5d30ff9640 10 monclient(hunting): my global_id is 4338 2024-03-25T18:18:55.391 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.390+0000 7f5d30ff9640 1 -- 172.21.15.97:0/2136096934 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d34134e10 con 0x7f5d340a98f0 2024-03-25T18:18:55.393 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.392+0000 7f5d30ff9640 1 -- 172.21.15.97:0/2136096934 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2245460344 0 0) 0x7f5d20004eb0 con 0x7f5d3413f710 2024-03-25T18:18:55.394 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.392+0000 7f5d30ff9640 1 -- 172.21.15.97:0/2136096934 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5d340ab920 con 0x7f5d3413f710 2024-03-25T18:18:55.394 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.392+0000 7f5d30ff9640 1 -- 172.21.15.97:0/2136096934 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f5d28001a30 con 0x7f5d3413f710 2024-03-25T18:18:55.394 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.392+0000 7f5d30ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:55.394 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.392+0000 7f5d30ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:18:55.394 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.392+0000 7f5d30ff9640 10 monclient(hunting): dump: 2024-03-25T18:18:55.394 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:18:55.394 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:18:55.394 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:18:55.394 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:18:55.394 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:55.394 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:18:55.394 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:18:55.394 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:18:55.395 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:18:55.395 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:18:55.395 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.392+0000 7f5d30ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:18:55.402 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.401+0000 7f5d30ff9640 1 -- 172.21.15.97:0/2136096934 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 388965979 0 0) 0x7f5d34134e10 con 0x7f5d340a98f0 2024-03-25T18:18:55.402 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.401+0000 7f5d30ff9640 1 -- 172.21.15.97:0/2136096934 --> v1:172.21.15.97:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5d20004eb0 con 0x7f5d340a98f0 2024-03-25T18:18:55.402 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.401+0000 7f5d30ff9640 1 -- 172.21.15.97:0/2136096934 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f5d24001830 con 0x7f5d340a98f0 2024-03-25T18:18:55.402 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.401+0000 7f5d30ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:55.402 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.401+0000 7f5d30ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:18:55.402 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.401+0000 7f5d30ff9640 10 monclient(hunting): dump: 2024-03-25T18:18:55.402 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:18:55.402 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:18:55.403 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:18:55.403 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:18:55.403 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:55.403 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:18:55.403 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:18:55.403 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:18:55.403 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:18:55.403 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:18:55.408 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.406+0000 7f5d30ff9640 1 -- 172.21.15.97:0/2136096934 <== mon.1 v1:172.21.15.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 551480752 0 0) 0x7f5d340ab920 con 0x7f5d3413f710 2024-03-25T18:18:55.408 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.406+0000 7f5d30ff9640 1 -- 172.21.15.97:0/2136096934 >> v1:172.21.15.97:6790/0 conn(0x7f5d341430c0 legacy=0x7f5d34135660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:55.408 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.406+0000 7f5d30ff9640 1 -- 172.21.15.97:0/2136096934 >> v1:172.21.15.97:6789/0 conn(0x7f5d340a98f0 legacy=0x7f5d341385e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:55.408 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.406+0000 7f5d30ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:18:55.408 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.406+0000 7f5d30ff9640 1 monclient: found mon.b 2024-03-25T18:18:55.408 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.406+0000 7f5d30ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:55.408 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.406+0000 7f5d30ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:18:55.409 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.406+0000 7f5d30ff9640 1 -- 172.21.15.97:0/2136096934 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d34137840 con 0x7f5d3413f710 2024-03-25T18:18:55.409 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.406+0000 7f5d30ff9640 10 monclient: _finish_auth 0 2024-03-25T18:18:55.409 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.406+0000 7f5d30ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:18:55.409 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.406+0000 7f5d30ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:55.409 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.406+0000 7f5d3a486640 5 monclient: authenticate success, global_id 4336 2024-03-25T18:18:55.409 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.406+0000 7f5d3a486640 10 monclient: _renew_subs 2024-03-25T18:18:55.409 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.406+0000 7f5d3a486640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:18:55.409 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.406+0000 7f5d3a486640 1 -- 172.21.15.97:0/2136096934 --> v1:172.21.15.98:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d34137a90 con 0x7f5d3413f710 2024-03-25T18:18:55.409 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.406+0000 7f5d3a486640 10 client.4336.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:18:55.409 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.406+0000 7f5d3a486640 10 monclient: _renew_subs 2024-03-25T18:18:55.409 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.406+0000 7f5d3a486640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:18:55.409 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.406+0000 7f5d3a486640 1 -- 172.21.15.97:0/2136096934 --> v1:172.21.15.98:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d341347e0 con 0x7f5d3413f710 2024-03-25T18:18:55.409 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.406+0000 7f5d3a486640 1 librados: init done 2024-03-25T18:18:55.410 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.407+0000 7f5d3a486640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:55.410 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.407+0000 7f5d3a486640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:55.410 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.407+0000 7f5d3a486640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:18:55.410 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.407+0000 7f5d3a486640 1 -- 172.21.15.97:0/2136096934 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d34134fa0 con 0x7f5d3413f710 2024-03-25T18:18:55.684 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.682+0000 7f5d30ff9640 1 -- 172.21.15.97:0/2136096934 <== mon.1 v1:172.21.15.98:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5d28001c70 con 0x7f5d3413f710 2024-03-25T18:18:55.684 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.682+0000 7f5d30ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:55.684 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.682+0000 7f5d30ff9640 1 -- 172.21.15.97:0/2136096934 <== mon.1 v1:172.21.15.98:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f5d280038c0 con 0x7f5d3413f710 2024-03-25T18:18:55.684 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.682+0000 7f5d30ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:55.684 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.682+0000 7f5d30ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:18:55.684 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.682+0000 7f5d30ff9640 10 monclient: dump: 2024-03-25T18:18:55.685 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:18:55.685 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:18:55.685 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:18:55.685 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:18:55.685 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:55.685 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:18:55.685 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:18:55.685 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:18:55.685 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:18:55.685 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:18:55.685 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.682+0000 7f5d327fc640 4 set_mon_vals no callback set 2024-03-25T18:18:55.685 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.683+0000 7f5d30ff9640 1 -- 172.21.15.97:0/2136096934 <== mon.1 v1:172.21.15.98:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2377622137 0 0) 0x7f5d28003ae0 con 0x7f5d3413f710 2024-03-25T18:18:55.686 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.684+0000 7f5d33fff640 10 monclient: get_auth_request con 0x7f5d2006df70 auth_method 0 2024-03-25T18:18:55.686 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.684+0000 7f5d30ff9640 10 client.4336.objecter ms_handle_connect 0x7f5d2006df70 2024-03-25T18:18:55.689 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.688+0000 7f5d30ff9640 1 -- 172.21.15.97:0/2136096934 <== mon.1 v1:172.21.15.98:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 430711918 0 0) 0x7f5d28004460 con 0x7f5d3413f710 2024-03-25T18:18:55.689 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.688+0000 7f5d30ff9640 10 client.4336.objecter ms_dispatch 0x7f5d34000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:18:55.689 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.688+0000 7f5d30ff9640 3 client.4336.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:18:55.689 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.688+0000 7f5d30ff9640 3 client.4336.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:18:55.689 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.688+0000 7f5d30ff9640 20 client.4336.objecter dump_active .. 0 homeless 2024-03-25T18:18:55.689 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.688+0000 7f5d30ff9640 1 -- 172.21.15.97:0/2136096934 <== mon.1 v1:172.21.15.98:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5d28050210 con 0x7f5d3413f710 2024-03-25T18:18:55.689 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.688+0000 7f5d30ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:55.689 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.688+0000 7f5d30ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:18:55.913 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.912+0000 7f5d3a486640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:55.913 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.912+0000 7f5d3a486640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:55.913 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.912+0000 7f5d3a486640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:18:55.913 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.912+0000 7f5d3a486640 1 -- 172.21.15.97:0/2136096934 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5d340627b0 con 0x7f5d3413f710 2024-03-25T18:18:55.943 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.942+0000 7f5d30ff9640 1 -- 172.21.15.97:0/2136096934 <== mon.1 v1:172.21.15.98:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2169427412) 0x7f5d28052140 con 0x7f5d3413f710 2024-03-25T18:18:55.943 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.942+0000 7f5d30ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:55.943 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.942+0000 7f5d30ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:18:55.943 INFO:teuthology.misc.health.smithi097.stdout: 2024-03-25T18:18:55.943 INFO:teuthology.misc.health.smithi097.stdout:{"epoch":10,"fsid":"4c3d472c-79b8-486b-9e3d-2738d4306754","created":"2024-03-25T18:16:14.737309+0000","modified":"2024-03-25T18:16:20.357492+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:20.357492+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cb5152b9-262f-45c3-94b8-18d0e81df558","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6daf8793-0838-473d-a7d6-9a0d2894c75d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"136f1493-3b9c-4f43-a2e7-641482d69dae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"397d538c-4f64-4daf-9e8b-662d09f91327","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4f1df08-663f-4bda-93f8-3878ab7c24b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c4b18d70-a3ce-4b15-9b70-eae5525bd761","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8dd4033a-de0e-4f8a-96dc-59f3c1f825b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0077722e-ffa6-4a3d-9891-929d9869b286","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:18:55.946 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.944+0000 7f5d3a486640 10 librados: watch_flush enter 2024-03-25T18:18:55.946 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.944+0000 7f5d3a486640 10 librados: watch_flush exit 2024-03-25T18:18:55.946 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.944+0000 7f5d3a486640 20 client.4336.objecter shutdown clearing up homeless session... 2024-03-25T18:18:55.946 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.944+0000 7f5d3a486640 10 client.4336.objecter successfully canceled tick 2024-03-25T18:18:55.946 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.945+0000 7f5d3a486640 1 -- 172.21.15.97:0/2136096934 >> v1:172.21.15.98:6800/800387573 conn(0x7f5d2006df70 legacy=0x7f5d20070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:55.946 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.945+0000 7f5d3a486640 10 monclient: shutdown 2024-03-25T18:18:55.946 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.945+0000 7f5d3a486640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:55.947 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.945+0000 7f5d3a486640 1 -- 172.21.15.97:0/2136096934 >> v1:172.21.15.98:6789/0 conn(0x7f5d3413f710 legacy=0x7f5d34133ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:55.947 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.945+0000 7f5d3a486640 1 -- 172.21.15.97:0/2136096934 shutdown_connections 2024-03-25T18:18:55.947 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.945+0000 7f5d3a486640 1 -- 172.21.15.97:0/2136096934 >> 172.21.15.97:0/2136096934 conn(0x7f5d340b7ea0 msgr2=0x7f5d3413ca40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:55.947 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.945+0000 7f5d3a486640 1 -- 172.21.15.97:0/2136096934 shutdown_connections 2024-03-25T18:18:55.947 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.945+0000 7f5d3a486640 1 -- 172.21.15.97:0/2136096934 wait complete. 2024-03-25T18:18:55.947 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:18:55.945+0000 7f5d3a486640 1 librados: shutdown 2024-03-25T18:18:55.958 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:19:02.764 DEBUG:teuthology.orchestra.run.smithi097:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:19:02.879 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:02.878+0000 7f9075e4d640 10 monclient: get_monmap_and_config 2024-03-25T18:19:02.880 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:02.878+0000 7f9075e4d640 10 monclient: build_initial_monmap 2024-03-25T18:19:02.880 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:02.878+0000 7f9075e4d640 10 monclient: monmap: 2024-03-25T18:19:02.880 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:19:02.880 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:19:02.880 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:19:02.880022+0000 2024-03-25T18:19:02.880 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:19:02.880022+0000 2024-03-25T18:19:02.880 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:02.880 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:19:02.880 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:19:02.880 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T18:19:02.880 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-b 2024-03-25T18:19:02.880 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:19:02.881 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:02.879+0000 7f9075e4d640 1 Processor -- start 2024-03-25T18:19:02.881 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:02.879+0000 7f9075e4d640 1 -- start start 2024-03-25T18:19:02.882 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:02.879+0000 7f9075e4d640 10 monclient: init 2024-03-25T18:19:02.882 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:02.880+0000 7f9075e4d640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:02.882 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:02.880+0000 7f9075e4d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:19:02.882 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:02.880+0000 7f9075e4d640 10 monclient(hunting): picked mon.noname-c con 0x7f90700a98f0 addr v1:172.21.15.98:6789/0 2024-03-25T18:19:02.882 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:02.880+0000 7f9075e4d640 10 monclient(hunting): picked mon.noname-b con 0x7f907013f710 addr v1:172.21.15.97:6790/0 2024-03-25T18:19:02.882 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:02.880+0000 7f9075e4d640 10 monclient(hunting): picked mon.noname-a con 0x7f90701430c0 addr v1:172.21.15.97:6789/0 2024-03-25T18:19:02.882 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:02.880+0000 7f9075e4d640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9070142520 con 0x7f90701430c0 2024-03-25T18:19:02.882 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:02.880+0000 7f9075e4d640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90700b28a0 con 0x7f90700a98f0 2024-03-25T18:19:02.882 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:02.880+0000 7f9075e4d640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90700b2a30 con 0x7f907013f710 2024-03-25T18:19:02.882 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:02.880+0000 7f9075e4d640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:02.882 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:02.880+0000 7f9075e4d640 10 monclient(hunting): authenticate will time out at 7755.156278s 2024-03-25T18:19:02.894 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:02.893+0000 7f906ffff640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f90701430c0 0x7f907014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:57178/0 (socket says 172.21.15.97:57178) 2024-03-25T18:19:02.894 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:02.893+0000 7f906ffff640 1 -- 172.21.15.97:0/288122707 learned_addr learned my addr 172.21.15.97:0/288122707 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:19:02.901 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:02.900+0000 7f906e7fc640 1 -- 172.21.15.97:0/288122707 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3707767244 0 0) 0x7f90700b28a0 con 0x7f90700a98f0 2024-03-25T18:19:02.901 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:02.900+0000 7f906e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:19:02.901 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:02.900+0000 7f906e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:02.902 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:02.900+0000 7f906e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:02.902 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:02.900+0000 7f906e7fc640 10 monclient(hunting): my global_id is 4339 2024-03-25T18:19:02.902 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:02.900+0000 7f906e7fc640 1 -- 172.21.15.97:0/288122707 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9058003520 con 0x7f90700a98f0 2024-03-25T18:19:02.908 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:02.907+0000 7f906e7fc640 1 -- 172.21.15.97:0/288122707 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2289674081 0 0) 0x7f90700b2a30 con 0x7f907013f710 2024-03-25T18:19:02.908 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:02.907+0000 7f906e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:19:02.908 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:02.907+0000 7f906e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:02.908 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:02.907+0000 7f906e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:02.908 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:02.907+0000 7f906e7fc640 10 monclient(hunting): my global_id is 4334 2024-03-25T18:19:02.908 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:02.907+0000 7f906e7fc640 1 -- 172.21.15.97:0/288122707 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90700b28a0 con 0x7f907013f710 2024-03-25T18:19:02.911 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:02.909+0000 7f906e7fc640 1 -- 172.21.15.97:0/288122707 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 547890264 0 0) 0x7f9070142520 con 0x7f90701430c0 2024-03-25T18:19:02.911 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:02.909+0000 7f906e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:19:02.911 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:02.909+0000 7f906e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:02.911 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:02.909+0000 7f906e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:02.911 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:02.909+0000 7f906e7fc640 10 monclient(hunting): my global_id is 4341 2024-03-25T18:19:02.911 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:02.909+0000 7f906e7fc640 1 -- 172.21.15.97:0/288122707 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90700b2a30 con 0x7f90701430c0 2024-03-25T18:19:02.916 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:02.914+0000 7f906e7fc640 1 -- 172.21.15.97:0/288122707 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3775609890 0 0) 0x7f9058003520 con 0x7f90700a98f0 2024-03-25T18:19:02.916 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:02.914+0000 7f906e7fc640 1 -- 172.21.15.97:0/288122707 >> v1:172.21.15.97:6790/0 conn(0x7f907013f710 legacy=0x7f90701429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:02.916 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:02.914+0000 7f906e7fc640 1 -- 172.21.15.97:0/288122707 >> v1:172.21.15.97:6789/0 conn(0x7f90701430c0 legacy=0x7f907014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:02.916 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:02.914+0000 7f906e7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:19:02.916 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:02.914+0000 7f906e7fc640 1 monclient: found mon.noname-c 2024-03-25T18:19:02.916 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:02.914+0000 7f906e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:02.916 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:02.914+0000 7f906e7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.98:6789/0 2024-03-25T18:19:02.916 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:02.914+0000 7f906e7fc640 1 -- 172.21.15.97:0/288122707 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90700b44c0 con 0x7f90700a98f0 2024-03-25T18:19:02.916 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:02.914+0000 7f906e7fc640 10 monclient: _finish_auth 0 2024-03-25T18:19:02.916 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:02.914+0000 7f906e7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:19:02.916 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:02.914+0000 7f906e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:02.916 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:02.914+0000 7f906e7fc640 1 -- 172.21.15.97:0/288122707 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f9060001690 con 0x7f90700a98f0 2024-03-25T18:19:02.916 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:02.914+0000 7f9075e4d640 5 monclient: authenticate success, global_id 4339 2024-03-25T18:19:02.916 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:02.914+0000 7f906e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:02.916 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:02.914+0000 7f906e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:19:02.917 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:02.914+0000 7f906e7fc640 10 monclient: dump: 2024-03-25T18:19:02.917 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:19:02.917 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:19:02.917 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:19:02.917 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:19:02.917 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:02.917 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:19:02.917 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:19:02.917 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:19:02.917 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:19:02.917 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:19:02.917 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:02.914+0000 7f9075e4d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:02.926 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:02.925+0000 7f906e7fc640 1 -- 172.21.15.97:0/288122707 <== mon.1 v1:172.21.15.98:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f90600017f0 con 0x7f90700a98f0 2024-03-25T18:19:02.926 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:02.925+0000 7f906e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:02.926 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:02.925+0000 7f906e7fc640 1 -- 172.21.15.97:0/288122707 <== mon.1 v1:172.21.15.98:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f9060001c00 con 0x7f90700a98f0 2024-03-25T18:19:02.926 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:02.925+0000 7f9075e4d640 10 monclient: get_monmap_and_config success 2024-03-25T18:19:02.926 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:02.925+0000 7f9075e4d640 4 set_mon_vals no callback set 2024-03-25T18:19:02.926 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:02.925+0000 7f906e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:02.926 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:02.925+0000 7f906e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:19:02.926 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:02.925+0000 7f906e7fc640 10 monclient: dump: 2024-03-25T18:19:02.926 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:19:02.927 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:19:02.927 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:19:02.927 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:19:02.927 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:02.927 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:19:02.927 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:19:02.927 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:19:02.927 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:19:02.927 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:19:02.928 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:02.925+0000 7f9075e4d640 10 monclient: shutdown 2024-03-25T18:19:02.928 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:02.925+0000 7f9075e4d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:02.928 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:02.925+0000 7f9075e4d640 1 -- 172.21.15.97:0/288122707 >> v1:172.21.15.98:6789/0 conn(0x7f90700a98f0 legacy=0x7f90700a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:02.928 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:02.926+0000 7f9075e4d640 1 -- 172.21.15.97:0/288122707 shutdown_connections 2024-03-25T18:19:02.928 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:02.926+0000 7f9075e4d640 1 -- 172.21.15.97:0/288122707 >> 172.21.15.97:0/288122707 conn(0x7f90700b7ea0 msgr2=0x7f90700b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:02.929 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:02.926+0000 7f9075e4d640 1 -- 172.21.15.97:0/288122707 shutdown_connections 2024-03-25T18:19:02.929 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:02.926+0000 7f9075e4d640 1 -- 172.21.15.97:0/288122707 wait complete. 2024-03-25T18:19:02.929 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:02.927+0000 7f9075e4d640 10 monclient: build_initial_monmap 2024-03-25T18:19:02.929 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:02.927+0000 7f9075e4d640 10 monclient: monmap: 2024-03-25T18:19:02.929 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:19:02.929 INFO:teuthology.misc.health.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:19:02.929 INFO:teuthology.misc.health.smithi097.stderr:last_changed 0.000000 2024-03-25T18:19:02.929 INFO:teuthology.misc.health.smithi097.stderr:created 0.000000 2024-03-25T18:19:02.929 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:02.929 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 1 2024-03-25T18:19:02.929 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:19:02.929 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T18:19:02.929 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-c 2024-03-25T18:19:02.929 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:19:02.929 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:02.927+0000 7f9075e4d640 1 librados: starting msgr at 2024-03-25T18:19:02.929 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:02.927+0000 7f9075e4d640 1 librados: starting objecter 2024-03-25T18:19:02.930 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:02.927+0000 7f9075e4d640 1 Processor -- start 2024-03-25T18:19:02.931 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:02.927+0000 7f9075e4d640 1 -- start start 2024-03-25T18:19:02.931 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:02.927+0000 7f9075e4d640 1 librados: setting wanted keys 2024-03-25T18:19:02.931 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:02.927+0000 7f9075e4d640 1 librados: calling monclient init 2024-03-25T18:19:02.931 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:02.927+0000 7f9075e4d640 10 monclient: init 2024-03-25T18:19:02.931 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:02.928+0000 7f9075e4d640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:02.931 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:02.928+0000 7f9075e4d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:19:02.931 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:02.928+0000 7f9075e4d640 10 monclient(hunting): picked mon.noname-b con 0x7f90700a98f0 addr v1:172.21.15.98:6789/0 2024-03-25T18:19:02.931 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:02.928+0000 7f9075e4d640 10 monclient(hunting): picked mon.noname-a con 0x7f907013f710 addr v1:172.21.15.97:6789/0 2024-03-25T18:19:02.931 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:02.928+0000 7f9075e4d640 10 monclient(hunting): picked mon.noname-c con 0x7f90701430c0 addr v1:172.21.15.97:6790/0 2024-03-25T18:19:02.931 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:02.928+0000 7f9075e4d640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9070138250 con 0x7f907013f710 2024-03-25T18:19:02.931 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:02.928+0000 7f9075e4d640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9070137d50 con 0x7f90700a98f0 2024-03-25T18:19:02.931 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:02.928+0000 7f9075e4d640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9070137ee0 con 0x7f90701430c0 2024-03-25T18:19:02.932 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:02.928+0000 7f9075e4d640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:02.932 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:02.928+0000 7f9075e4d640 10 monclient(hunting): authenticate will time out at 7755.204484s 2024-03-25T18:19:02.938 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:02.937+0000 7f906f7fe640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f90700a98f0 0x7f9070137640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.97:52830/0 (socket says 172.21.15.97:52830) 2024-03-25T18:19:02.938 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:02.937+0000 7f906f7fe640 1 -- 172.21.15.97:0/2359729845 learned_addr learned my addr 172.21.15.97:0/2359729845 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:19:02.939 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:02.938+0000 7f9057fff640 10 client.?.objecter ms_handle_connect 0x7f90700a98f0 2024-03-25T18:19:02.939 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:02.938+0000 7f9057fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:02.948 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:02.947+0000 7f9057fff640 10 client.?.objecter ms_handle_connect 0x7f907013f710 2024-03-25T18:19:02.948 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:02.947+0000 7f9057fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:02.951 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:02.950+0000 7f9057fff640 1 -- 172.21.15.97:0/2359729845 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 383659002 0 0) 0x7f9070137d50 con 0x7f90700a98f0 2024-03-25T18:19:02.951 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:02.950+0000 7f9057fff640 10 monclient(hunting): _negotiate 2024-03-25T18:19:02.951 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:02.950+0000 7f9057fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:02.951 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:02.950+0000 7f9057fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:02.951 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:02.950+0000 7f9057fff640 10 monclient(hunting): my global_id is 4342 2024-03-25T18:19:02.952 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:02.950+0000 7f9057fff640 1 -- 172.21.15.97:0/2359729845 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f905c004eb0 con 0x7f90700a98f0 2024-03-25T18:19:02.953 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:02.952+0000 7f9057fff640 10 client.?.objecter ms_handle_connect 0x7f90701430c0 2024-03-25T18:19:02.953 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:02.952+0000 7f9057fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:02.966 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:02.965+0000 7f9057fff640 1 -- 172.21.15.97:0/2359729845 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2562943181 0 0) 0x7f905c004eb0 con 0x7f90700a98f0 2024-03-25T18:19:02.966 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:02.965+0000 7f9057fff640 1 -- 172.21.15.97:0/2359729845 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9070137d50 con 0x7f90700a98f0 2024-03-25T18:19:02.966 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:02.965+0000 7f9057fff640 1 -- 172.21.15.97:0/2359729845 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f9060001bc0 con 0x7f90700a98f0 2024-03-25T18:19:02.966 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:02.965+0000 7f9057fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:02.966 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:02.965+0000 7f9057fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:19:02.966 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:02.965+0000 7f9057fff640 10 monclient(hunting): dump: 2024-03-25T18:19:02.966 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:19:02.967 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:19:02.967 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:19:02.967 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:19:02.967 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:02.967 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:19:02.967 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:19:02.967 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:19:02.967 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:19:02.967 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:19:02.967 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:02.965+0000 7f9057fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:19:02.969 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:02.967+0000 7f9057fff640 1 -- 172.21.15.97:0/2359729845 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 36722367 0 0) 0x7f9070137ee0 con 0x7f90701430c0 2024-03-25T18:19:02.969 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:02.968+0000 7f9057fff640 10 monclient(hunting): _negotiate 2024-03-25T18:19:02.969 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:02.968+0000 7f9057fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:02.969 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:02.968+0000 7f9057fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:02.969 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:02.968+0000 7f9057fff640 10 monclient(hunting): my global_id is 4337 2024-03-25T18:19:02.969 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:02.968+0000 7f9057fff640 1 -- 172.21.15.97:0/2359729845 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f905c004eb0 con 0x7f90701430c0 2024-03-25T18:19:02.974 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:02.972+0000 7f9057fff640 1 -- 172.21.15.97:0/2359729845 <== mon.1 v1:172.21.15.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2604433665 0 0) 0x7f9070137d50 con 0x7f90700a98f0 2024-03-25T18:19:02.974 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:02.972+0000 7f9057fff640 1 -- 172.21.15.97:0/2359729845 >> v1:172.21.15.97:6790/0 conn(0x7f90701430c0 legacy=0x7f90701d7df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:02.974 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:02.972+0000 7f9057fff640 1 -- 172.21.15.97:0/2359729845 >> v1:172.21.15.97:6789/0 conn(0x7f907013f710 legacy=0x7f90701d76e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:02.974 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:02.972+0000 7f9057fff640 10 monclient: _finish_hunting 0 2024-03-25T18:19:02.974 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:02.972+0000 7f9057fff640 1 monclient: found mon.b 2024-03-25T18:19:02.974 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:02.972+0000 7f9057fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:02.974 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:02.972+0000 7f9057fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:19:02.974 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:02.972+0000 7f9057fff640 1 -- 172.21.15.97:0/2359729845 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90700b7730 con 0x7f90700a98f0 2024-03-25T18:19:02.974 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:02.972+0000 7f9057fff640 10 monclient: _finish_auth 0 2024-03-25T18:19:02.974 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:02.972+0000 7f9057fff640 10 monclient: _check_auth_tickets 2024-03-25T18:19:02.975 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:02.972+0000 7f9057fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:02.975 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:02.972+0000 7f9075e4d640 5 monclient: authenticate success, global_id 4342 2024-03-25T18:19:02.975 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:02.972+0000 7f9075e4d640 10 monclient: _renew_subs 2024-03-25T18:19:02.975 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:02.972+0000 7f9075e4d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:19:02.975 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:02.972+0000 7f9075e4d640 1 -- 172.21.15.97:0/2359729845 --> v1:172.21.15.98:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90700b7920 con 0x7f90700a98f0 2024-03-25T18:19:02.975 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:02.973+0000 7f9075e4d640 10 client.4342.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:19:02.975 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:02.973+0000 7f9075e4d640 10 monclient: _renew_subs 2024-03-25T18:19:02.975 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:02.973+0000 7f9075e4d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:19:02.975 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:02.973+0000 7f9075e4d640 1 -- 172.21.15.97:0/2359729845 --> v1:172.21.15.98:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f90701e33d0 con 0x7f90700a98f0 2024-03-25T18:19:02.975 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:02.973+0000 7f9075e4d640 1 librados: init done 2024-03-25T18:19:02.976 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:02.974+0000 7f9075e4d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:02.976 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:02.974+0000 7f9075e4d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:02.976 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:02.974+0000 7f9075e4d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:19:02.976 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:02.974+0000 7f9075e4d640 1 -- 172.21.15.97:0/2359729845 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90700627b0 con 0x7f90700a98f0 2024-03-25T18:19:03.247 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:03.246+0000 7f9057fff640 1 -- 172.21.15.97:0/2359729845 <== mon.1 v1:172.21.15.98:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f90600011f0 con 0x7f90700a98f0 2024-03-25T18:19:03.247 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:03.246+0000 7f9057fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:03.247 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:03.246+0000 7f9057fff640 1 -- 172.21.15.97:0/2359729845 <== mon.1 v1:172.21.15.98:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f9060003b00 con 0x7f90700a98f0 2024-03-25T18:19:03.248 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:03.246+0000 7f9057fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:03.248 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:03.246+0000 7f9057fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:19:03.248 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:03.246+0000 7f9057fff640 10 monclient: dump: 2024-03-25T18:19:03.248 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:19:03.248 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:19:03.248 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:19:03.248 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:19:03.248 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:03.248 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:19:03.248 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:19:03.248 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:19:03.248 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:19:03.248 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:19:03.249 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:03.246+0000 7f906dffb640 4 set_mon_vals no callback set 2024-03-25T18:19:03.249 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:03.247+0000 7f9057fff640 1 -- 172.21.15.97:0/2359729845 <== mon.1 v1:172.21.15.98:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2377622137 0 0) 0x7f9060003d20 con 0x7f90700a98f0 2024-03-25T18:19:03.249 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:03.248+0000 7f906effd640 10 monclient: get_auth_request con 0x7f905c06dd40 auth_method 0 2024-03-25T18:19:03.249 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:03.248+0000 7f9057fff640 10 client.4342.objecter ms_handle_connect 0x7f905c06dd40 2024-03-25T18:19:03.252 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:03.251+0000 7f9057fff640 1 -- 172.21.15.97:0/2359729845 <== mon.1 v1:172.21.15.98:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 430711918 0 0) 0x7f9060054920 con 0x7f90700a98f0 2024-03-25T18:19:03.252 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:03.251+0000 7f9057fff640 10 client.4342.objecter ms_dispatch 0x7f9070000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:19:03.252 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:03.251+0000 7f9057fff640 3 client.4342.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:19:03.253 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:03.251+0000 7f9057fff640 3 client.4342.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:19:03.253 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:03.251+0000 7f9057fff640 20 client.4342.objecter dump_active .. 0 homeless 2024-03-25T18:19:03.253 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:03.251+0000 7f9057fff640 1 -- 172.21.15.97:0/2359729845 <== mon.1 v1:172.21.15.98:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9060050260 con 0x7f90700a98f0 2024-03-25T18:19:03.253 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:03.251+0000 7f9057fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:03.253 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:03.251+0000 7f9057fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:19:03.503 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:03.502+0000 7f9075e4d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:03.503 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:03.502+0000 7f9075e4d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:03.503 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:03.502+0000 7f9075e4d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:19:03.503 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:03.502+0000 7f9075e4d640 1 -- 172.21.15.97:0/2359729845 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f907013df50 con 0x7f90700a98f0 2024-03-25T18:19:03.526 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:03.525+0000 7f9057fff640 1 -- 172.21.15.97:0/2359729845 <== mon.1 v1:172.21.15.98:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2169427412) 0x7f9060052360 con 0x7f90700a98f0 2024-03-25T18:19:03.526 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:03.525+0000 7f9057fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:03.527 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:03.525+0000 7f9057fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:19:03.527 INFO:teuthology.misc.health.smithi097.stdout: 2024-03-25T18:19:03.527 INFO:teuthology.misc.health.smithi097.stdout:{"epoch":10,"fsid":"4c3d472c-79b8-486b-9e3d-2738d4306754","created":"2024-03-25T18:16:14.737309+0000","modified":"2024-03-25T18:16:20.357492+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:20.357492+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cb5152b9-262f-45c3-94b8-18d0e81df558","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6daf8793-0838-473d-a7d6-9a0d2894c75d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"136f1493-3b9c-4f43-a2e7-641482d69dae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"397d538c-4f64-4daf-9e8b-662d09f91327","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4f1df08-663f-4bda-93f8-3878ab7c24b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c4b18d70-a3ce-4b15-9b70-eae5525bd761","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8dd4033a-de0e-4f8a-96dc-59f3c1f825b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0077722e-ffa6-4a3d-9891-929d9869b286","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:19:03.529 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:03.527+0000 7f9075e4d640 10 librados: watch_flush enter 2024-03-25T18:19:03.529 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:03.527+0000 7f9075e4d640 10 librados: watch_flush exit 2024-03-25T18:19:03.529 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:03.527+0000 7f9075e4d640 20 client.4342.objecter shutdown clearing up homeless session... 2024-03-25T18:19:03.530 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:03.527+0000 7f9075e4d640 10 client.4342.objecter successfully canceled tick 2024-03-25T18:19:03.530 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:03.527+0000 7f9075e4d640 1 -- 172.21.15.97:0/2359729845 >> v1:172.21.15.98:6800/800387573 conn(0x7f905c06dd40 legacy=0x7f905c070200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:03.530 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:03.527+0000 7f9075e4d640 10 monclient: shutdown 2024-03-25T18:19:03.530 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:03.527+0000 7f9075e4d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:03.530 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:03.528+0000 7f9075e4d640 1 -- 172.21.15.97:0/2359729845 >> v1:172.21.15.98:6789/0 conn(0x7f90700a98f0 legacy=0x7f9070137640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:03.530 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:03.528+0000 7f9075e4d640 1 -- 172.21.15.97:0/2359729845 shutdown_connections 2024-03-25T18:19:03.530 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:03.528+0000 7f9075e4d640 1 -- 172.21.15.97:0/2359729845 >> 172.21.15.97:0/2359729845 conn(0x7f90700b7ea0 msgr2=0x7f9070074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:03.530 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:03.528+0000 7f9075e4d640 1 -- 172.21.15.97:0/2359729845 shutdown_connections 2024-03-25T18:19:03.530 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:03.528+0000 7f9075e4d640 1 -- 172.21.15.97:0/2359729845 wait complete. 2024-03-25T18:19:03.530 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:03.528+0000 7f9075e4d640 1 librados: shutdown 2024-03-25T18:19:03.540 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:19:10.348 DEBUG:teuthology.orchestra.run.smithi097:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:19:10.466 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.464+0000 7f5ae4d54640 10 monclient: get_monmap_and_config 2024-03-25T18:19:10.466 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.464+0000 7f5ae4d54640 10 monclient: build_initial_monmap 2024-03-25T18:19:10.466 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.464+0000 7f5ae4d54640 10 monclient: monmap: 2024-03-25T18:19:10.466 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:19:10.466 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:19:10.466 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:19:10.465920+0000 2024-03-25T18:19:10.466 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:19:10.465920+0000 2024-03-25T18:19:10.466 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:10.466 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:19:10.466 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:19:10.466 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T18:19:10.467 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-b 2024-03-25T18:19:10.467 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:19:10.468 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.465+0000 7f5ae4d54640 1 Processor -- start 2024-03-25T18:19:10.468 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.465+0000 7f5ae4d54640 1 -- start start 2024-03-25T18:19:10.468 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.465+0000 7f5ae4d54640 10 monclient: init 2024-03-25T18:19:10.468 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.466+0000 7f5ae4d54640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:10.468 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.466+0000 7f5ae4d54640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:19:10.468 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.466+0000 7f5ae4d54640 10 monclient(hunting): picked mon.noname-a con 0x7f5ae00b1ef0 addr v1:172.21.15.97:6789/0 2024-03-25T18:19:10.469 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.466+0000 7f5ae4d54640 10 monclient(hunting): picked mon.noname-c con 0x7f5ae01473a0 addr v1:172.21.15.98:6789/0 2024-03-25T18:19:10.469 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.466+0000 7f5ae4d54640 10 monclient(hunting): picked mon.noname-b con 0x7f5ae0150070 addr v1:172.21.15.97:6790/0 2024-03-25T18:19:10.469 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.466+0000 7f5ae4d54640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ae00b2570 con 0x7f5ae00b1ef0 2024-03-25T18:19:10.469 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.466+0000 7f5ae4d54640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ae0146c20 con 0x7f5ae01473a0 2024-03-25T18:19:10.469 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.466+0000 7f5ae4d54640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ae0146e80 con 0x7f5ae0150070 2024-03-25T18:19:10.469 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.466+0000 7f5ae4d54640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:10.469 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.466+0000 7f5ae4d54640 10 monclient(hunting): authenticate will time out at 7762.742312s 2024-03-25T18:19:10.477 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.476+0000 7f5addd9b640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f5ae01473a0 0x7f5ae0143b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.97:57612/0 (socket says 172.21.15.97:57612) 2024-03-25T18:19:10.477 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.476+0000 7f5addd9b640 1 -- 172.21.15.97:0/271870224 learned_addr learned my addr 172.21.15.97:0/271870224 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:19:10.492 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.490+0000 7f5add59a640 1 -- 172.21.15.97:0/271870224 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2908700105 0 0) 0x7f5ae00b2570 con 0x7f5ae00b1ef0 2024-03-25T18:19:10.492 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.490+0000 7f5add59a640 10 monclient(hunting): _negotiate 2024-03-25T18:19:10.492 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.490+0000 7f5add59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:10.492 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.490+0000 7f5add59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:10.492 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.490+0000 7f5add59a640 10 monclient(hunting): my global_id is 4347 2024-03-25T18:19:10.492 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.490+0000 7f5add59a640 1 -- 172.21.15.97:0/271870224 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ac8003520 con 0x7f5ae00b1ef0 2024-03-25T18:19:10.492 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.491+0000 7f5add59a640 1 -- 172.21.15.97:0/271870224 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1210573317 0 0) 0x7f5ae0146e80 con 0x7f5ae0150070 2024-03-25T18:19:10.493 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.491+0000 7f5add59a640 10 monclient(hunting): _negotiate 2024-03-25T18:19:10.493 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.491+0000 7f5add59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:10.493 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.491+0000 7f5add59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:10.493 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.491+0000 7f5add59a640 10 monclient(hunting): my global_id is 4340 2024-03-25T18:19:10.493 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.491+0000 7f5add59a640 1 -- 172.21.15.97:0/271870224 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ae00b2570 con 0x7f5ae0150070 2024-03-25T18:19:10.498 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.497+0000 7f5add59a640 1 -- 172.21.15.97:0/271870224 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2199775758 0 0) 0x7f5ae0146c20 con 0x7f5ae01473a0 2024-03-25T18:19:10.498 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.497+0000 7f5add59a640 10 monclient(hunting): _negotiate 2024-03-25T18:19:10.498 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.497+0000 7f5add59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:10.498 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.497+0000 7f5add59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:10.498 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.497+0000 7f5add59a640 10 monclient(hunting): my global_id is 4345 2024-03-25T18:19:10.498 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.497+0000 7f5add59a640 1 -- 172.21.15.97:0/271870224 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ae0146e80 con 0x7f5ae01473a0 2024-03-25T18:19:10.505 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.504+0000 7f5add59a640 1 -- 172.21.15.97:0/271870224 <== mon.2 v1:172.21.15.97:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2530943169 0 0) 0x7f5ae00b2570 con 0x7f5ae0150070 2024-03-25T18:19:10.505 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.504+0000 7f5add59a640 1 -- 172.21.15.97:0/271870224 >> v1:172.21.15.98:6789/0 conn(0x7f5ae01473a0 legacy=0x7f5ae0143b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:10.505 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.504+0000 7f5add59a640 1 -- 172.21.15.97:0/271870224 >> v1:172.21.15.97:6789/0 conn(0x7f5ae00b1ef0 legacy=0x7f5ae00b28a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:10.506 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.504+0000 7f5add59a640 10 monclient: _finish_hunting 0 2024-03-25T18:19:10.506 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.504+0000 7f5add59a640 1 monclient: found mon.noname-b 2024-03-25T18:19:10.506 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.504+0000 7f5add59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:10.506 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.504+0000 7f5add59a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.97:6790/0 2024-03-25T18:19:10.506 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.504+0000 7f5add59a640 1 -- 172.21.15.97:0/271870224 --> v1:172.21.15.97:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ae006e1b0 con 0x7f5ae0150070 2024-03-25T18:19:10.506 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.504+0000 7f5add59a640 10 monclient: _finish_auth 0 2024-03-25T18:19:10.506 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.504+0000 7f5add59a640 10 monclient: _check_auth_tickets 2024-03-25T18:19:10.506 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.504+0000 7f5add59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:10.506 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.504+0000 7f5add59a640 1 -- 172.21.15.97:0/271870224 <== mon.2 v1:172.21.15.97:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f5ad80019f0 con 0x7f5ae0150070 2024-03-25T18:19:10.506 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.504+0000 7f5add59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:10.506 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.504+0000 7f5add59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:19:10.507 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.504+0000 7f5add59a640 10 monclient: dump: 2024-03-25T18:19:10.507 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:19:10.507 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:19:10.507 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:19:10.507 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:19:10.507 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:10.507 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:19:10.507 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:19:10.507 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:19:10.507 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:19:10.507 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:19:10.507 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.504+0000 7f5ae4d54640 5 monclient: authenticate success, global_id 4340 2024-03-25T18:19:10.507 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.504+0000 7f5ae4d54640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:10.519 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.517+0000 7f5add59a640 1 -- 172.21.15.97:0/271870224 <== mon.2 v1:172.21.15.97:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5ad8001b50 con 0x7f5ae0150070 2024-03-25T18:19:10.519 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.517+0000 7f5add59a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:10.529 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.517+0000 7f5add59a640 1 -- 172.21.15.97:0/271870224 <== mon.2 v1:172.21.15.97:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f5ad8001f60 con 0x7f5ae0150070 2024-03-25T18:19:10.529 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.517+0000 7f5add59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:10.529 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.517+0000 7f5add59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:19:10.529 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.517+0000 7f5add59a640 10 monclient: dump: 2024-03-25T18:19:10.530 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:19:10.530 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:19:10.530 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:19:10.530 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:19:10.530 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:10.530 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:19:10.530 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:19:10.530 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:19:10.530 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:19:10.530 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:19:10.531 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.517+0000 7f5ae4d54640 10 monclient: get_monmap_and_config success 2024-03-25T18:19:10.531 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.517+0000 7f5ae4d54640 4 set_mon_vals no callback set 2024-03-25T18:19:10.531 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.518+0000 7f5ae4d54640 10 monclient: shutdown 2024-03-25T18:19:10.531 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.518+0000 7f5ae4d54640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:10.531 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.518+0000 7f5ae4d54640 1 -- 172.21.15.97:0/271870224 >> v1:172.21.15.97:6790/0 conn(0x7f5ae0150070 legacy=0x7f5ae0152460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:10.531 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.518+0000 7f5ae4d54640 1 -- 172.21.15.97:0/271870224 shutdown_connections 2024-03-25T18:19:10.531 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.518+0000 7f5ae4d54640 1 -- 172.21.15.97:0/271870224 >> 172.21.15.97:0/271870224 conn(0x7f5ae00b7ea0 msgr2=0x7f5ae00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:10.531 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.518+0000 7f5ae4d54640 1 -- 172.21.15.97:0/271870224 shutdown_connections 2024-03-25T18:19:10.531 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.518+0000 7f5ae4d54640 1 -- 172.21.15.97:0/271870224 wait complete. 2024-03-25T18:19:10.533 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.529+0000 7f5ae4d54640 10 monclient: build_initial_monmap 2024-03-25T18:19:10.533 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.529+0000 7f5ae4d54640 10 monclient: monmap: 2024-03-25T18:19:10.533 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:19:10.533 INFO:teuthology.misc.health.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:19:10.533 INFO:teuthology.misc.health.smithi097.stderr:last_changed 0.000000 2024-03-25T18:19:10.533 INFO:teuthology.misc.health.smithi097.stderr:created 0.000000 2024-03-25T18:19:10.533 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:10.533 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 1 2024-03-25T18:19:10.534 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:19:10.534 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T18:19:10.534 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-c 2024-03-25T18:19:10.534 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:19:10.534 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.529+0000 7f5ae4d54640 1 librados: starting msgr at 2024-03-25T18:19:10.534 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.529+0000 7f5ae4d54640 1 librados: starting objecter 2024-03-25T18:19:10.534 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.529+0000 7f5ae4d54640 1 Processor -- start 2024-03-25T18:19:10.534 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.529+0000 7f5ae4d54640 1 -- start start 2024-03-25T18:19:10.534 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.529+0000 7f5ae4d54640 1 librados: setting wanted keys 2024-03-25T18:19:10.534 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.529+0000 7f5ae4d54640 1 librados: calling monclient init 2024-03-25T18:19:10.534 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.529+0000 7f5ae4d54640 10 monclient: init 2024-03-25T18:19:10.535 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.530+0000 7f5ae4d54640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:10.535 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.530+0000 7f5ae4d54640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:19:10.535 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.530+0000 7f5ae4d54640 10 monclient(hunting): picked mon.noname-c con 0x7f5ae00b1ef0 addr v1:172.21.15.97:6790/0 2024-03-25T18:19:10.535 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.530+0000 7f5ae4d54640 10 monclient(hunting): picked mon.noname-b con 0x7f5ae01473a0 addr v1:172.21.15.98:6789/0 2024-03-25T18:19:10.535 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.530+0000 7f5ae4d54640 10 monclient(hunting): picked mon.noname-a con 0x7f5ae0150070 addr v1:172.21.15.97:6789/0 2024-03-25T18:19:10.535 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.530+0000 7f5ae4d54640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ae0144560 con 0x7f5ae0150070 2024-03-25T18:19:10.535 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.530+0000 7f5ae4d54640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ae01dbb20 con 0x7f5ae01473a0 2024-03-25T18:19:10.535 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.530+0000 7f5ae4d54640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ae01455e0 con 0x7f5ae00b1ef0 2024-03-25T18:19:10.535 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.530+0000 7f5ae4d54640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:10.535 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.530+0000 7f5ae4d54640 10 monclient(hunting): authenticate will time out at 7762.806687s 2024-03-25T18:19:10.545 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.543+0000 7f5aded9d640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f5ae0150070 0x7f5ae01e64c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:36138/0 (socket says 172.21.15.97:36138) 2024-03-25T18:19:10.545 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.543+0000 7f5aded9d640 1 -- 172.21.15.97:0/1097947061 learned_addr learned my addr 172.21.15.97:0/1097947061 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:19:10.547 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.546+0000 7f5aceffd640 10 client.?.objecter ms_handle_connect 0x7f5ae0150070 2024-03-25T18:19:10.548 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.546+0000 7f5aceffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:10.551 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.549+0000 7f5aceffd640 10 client.?.objecter ms_handle_connect 0x7f5ae01473a0 2024-03-25T18:19:10.551 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.549+0000 7f5aceffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:10.552 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.551+0000 7f5aceffd640 10 client.?.objecter ms_handle_connect 0x7f5ae00b1ef0 2024-03-25T18:19:10.552 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.551+0000 7f5aceffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:10.560 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.559+0000 7f5aceffd640 1 -- 172.21.15.97:0/1097947061 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1518215189 0 0) 0x7f5ae01dbb20 con 0x7f5ae01473a0 2024-03-25T18:19:10.560 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.559+0000 7f5aceffd640 10 monclient(hunting): _negotiate 2024-03-25T18:19:10.560 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.559+0000 7f5aceffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:10.560 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.559+0000 7f5aceffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:10.560 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.559+0000 7f5aceffd640 10 monclient(hunting): my global_id is 4348 2024-03-25T18:19:10.561 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.559+0000 7f5aceffd640 1 -- 172.21.15.97:0/1097947061 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ac0004eb0 con 0x7f5ae01473a0 2024-03-25T18:19:10.562 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.561+0000 7f5aceffd640 1 -- 172.21.15.97:0/1097947061 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3877035188 0 0) 0x7f5ae0144560 con 0x7f5ae0150070 2024-03-25T18:19:10.563 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.561+0000 7f5aceffd640 10 monclient(hunting): _negotiate 2024-03-25T18:19:10.563 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.561+0000 7f5aceffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:10.563 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.561+0000 7f5aceffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:10.563 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.561+0000 7f5aceffd640 10 monclient(hunting): my global_id is 4350 2024-03-25T18:19:10.563 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.561+0000 7f5aceffd640 1 -- 172.21.15.97:0/1097947061 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ae01dbb20 con 0x7f5ae0150070 2024-03-25T18:19:10.571 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.569+0000 7f5aceffd640 1 -- 172.21.15.97:0/1097947061 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4002783815 0 0) 0x7f5ae01455e0 con 0x7f5ae00b1ef0 2024-03-25T18:19:10.571 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.569+0000 7f5aceffd640 10 monclient(hunting): _negotiate 2024-03-25T18:19:10.571 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.569+0000 7f5aceffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:10.571 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.569+0000 7f5aceffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:10.571 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.569+0000 7f5aceffd640 10 monclient(hunting): my global_id is 4343 2024-03-25T18:19:10.571 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.569+0000 7f5aceffd640 1 -- 172.21.15.97:0/1097947061 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ae0144560 con 0x7f5ae00b1ef0 2024-03-25T18:19:10.589 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.587+0000 7f5aceffd640 1 -- 172.21.15.97:0/1097947061 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4066523555 0 0) 0x7f5ac0004eb0 con 0x7f5ae01473a0 2024-03-25T18:19:10.589 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.587+0000 7f5aceffd640 1 -- 172.21.15.97:0/1097947061 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5ae01455e0 con 0x7f5ae01473a0 2024-03-25T18:19:10.589 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.587+0000 7f5aceffd640 1 -- 172.21.15.97:0/1097947061 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f5ad4001930 con 0x7f5ae01473a0 2024-03-25T18:19:10.589 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.587+0000 7f5aceffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:10.589 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.587+0000 7f5aceffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:19:10.590 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.587+0000 7f5aceffd640 10 monclient(hunting): dump: 2024-03-25T18:19:10.590 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:19:10.590 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:19:10.590 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:19:10.590 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:19:10.590 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:10.590 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:19:10.590 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:19:10.590 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:19:10.590 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:19:10.591 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:19:10.591 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.587+0000 7f5aceffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:19:10.595 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.593+0000 7f5aceffd640 1 -- 172.21.15.97:0/1097947061 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 260058162 0 0) 0x7f5ae01dbb20 con 0x7f5ae0150070 2024-03-25T18:19:10.595 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.593+0000 7f5aceffd640 1 -- 172.21.15.97:0/1097947061 --> v1:172.21.15.97:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5ac0004eb0 con 0x7f5ae0150070 2024-03-25T18:19:10.595 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.593+0000 7f5aceffd640 1 -- 172.21.15.97:0/1097947061 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f5ad8001f20 con 0x7f5ae0150070 2024-03-25T18:19:10.595 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.593+0000 7f5aceffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:10.595 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.593+0000 7f5aceffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:19:10.595 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.593+0000 7f5aceffd640 10 monclient(hunting): dump: 2024-03-25T18:19:10.595 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:19:10.595 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:19:10.595 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:19:10.596 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:19:10.596 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:10.596 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:19:10.596 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:19:10.596 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:19:10.596 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:19:10.596 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:19:10.596 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.594+0000 7f5aceffd640 1 -- 172.21.15.97:0/1097947061 <== mon.2 v1:172.21.15.97:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3877004451 0 0) 0x7f5ae0144560 con 0x7f5ae00b1ef0 2024-03-25T18:19:10.597 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.594+0000 7f5aceffd640 1 -- 172.21.15.97:0/1097947061 --> v1:172.21.15.97:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5ae01dbb20 con 0x7f5ae00b1ef0 2024-03-25T18:19:10.597 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.594+0000 7f5aceffd640 1 -- 172.21.15.97:0/1097947061 <== mon.2 v1:172.21.15.97:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f5ad00017b0 con 0x7f5ae00b1ef0 2024-03-25T18:19:10.597 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.594+0000 7f5aceffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:10.597 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.594+0000 7f5aceffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:19:10.597 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.594+0000 7f5aceffd640 10 monclient(hunting): dump: 2024-03-25T18:19:10.597 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:19:10.597 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:19:10.597 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:19:10.597 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:19:10.597 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:10.597 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:19:10.598 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:19:10.598 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:19:10.598 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:19:10.598 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:19:10.599 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.597+0000 7f5aceffd640 1 -- 172.21.15.97:0/1097947061 <== mon.1 v1:172.21.15.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2672880405 0 0) 0x7f5ae01455e0 con 0x7f5ae01473a0 2024-03-25T18:19:10.599 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.597+0000 7f5aceffd640 1 -- 172.21.15.97:0/1097947061 >> v1:172.21.15.97:6790/0 conn(0x7f5ae00b1ef0 legacy=0x7f5ae01dc020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:10.600 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.597+0000 7f5aceffd640 1 -- 172.21.15.97:0/1097947061 >> v1:172.21.15.97:6789/0 conn(0x7f5ae0150070 legacy=0x7f5ae01e64c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:10.600 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.597+0000 7f5aceffd640 10 monclient: _finish_hunting 0 2024-03-25T18:19:10.600 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.597+0000 7f5aceffd640 1 monclient: found mon.b 2024-03-25T18:19:10.600 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.597+0000 7f5aceffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:10.600 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.597+0000 7f5aceffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:19:10.600 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.597+0000 7f5aceffd640 1 -- 172.21.15.97:0/1097947061 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ae0145770 con 0x7f5ae01473a0 2024-03-25T18:19:10.600 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.597+0000 7f5aceffd640 10 monclient: _finish_auth 0 2024-03-25T18:19:10.600 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.597+0000 7f5aceffd640 10 monclient: _check_auth_tickets 2024-03-25T18:19:10.600 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.597+0000 7f5aceffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:10.600 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.597+0000 7f5ae4d54640 5 monclient: authenticate success, global_id 4348 2024-03-25T18:19:10.600 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.597+0000 7f5ae4d54640 10 monclient: _renew_subs 2024-03-25T18:19:10.600 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.597+0000 7f5ae4d54640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:19:10.600 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.597+0000 7f5ae4d54640 1 -- 172.21.15.97:0/1097947061 --> v1:172.21.15.98:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ae01dc730 con 0x7f5ae01473a0 2024-03-25T18:19:10.601 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.598+0000 7f5ae4d54640 10 client.4348.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:19:10.601 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.598+0000 7f5ae4d54640 10 monclient: _renew_subs 2024-03-25T18:19:10.601 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.598+0000 7f5ae4d54640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:19:10.601 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.598+0000 7f5ae4d54640 1 -- 172.21.15.97:0/1097947061 --> v1:172.21.15.98:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ae00710f0 con 0x7f5ae01473a0 2024-03-25T18:19:10.601 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.598+0000 7f5ae4d54640 1 librados: init done 2024-03-25T18:19:10.601 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.599+0000 7f5ae4d54640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:10.601 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.599+0000 7f5ae4d54640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:10.602 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.599+0000 7f5ae4d54640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:19:10.602 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.599+0000 7f5ae4d54640 1 -- 172.21.15.97:0/1097947061 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ae01dbcb0 con 0x7f5ae01473a0 2024-03-25T18:19:10.870 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.869+0000 7f5aceffd640 1 -- 172.21.15.97:0/1097947061 <== mon.1 v1:172.21.15.98:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5ad40047e0 con 0x7f5ae01473a0 2024-03-25T18:19:10.870 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.869+0000 7f5aceffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:10.870 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.869+0000 7f5aceffd640 1 -- 172.21.15.97:0/1097947061 <== mon.1 v1:172.21.15.98:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f5ad4004950 con 0x7f5ae01473a0 2024-03-25T18:19:10.870 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.869+0000 7f5aceffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:10.871 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.869+0000 7f5aceffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:19:10.871 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.869+0000 7f5aceffd640 10 monclient: dump: 2024-03-25T18:19:10.871 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:19:10.871 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:19:10.871 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:19:10.871 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:19:10.871 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:10.871 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:19:10.871 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:19:10.871 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:19:10.871 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:19:10.871 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:19:10.871 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.869+0000 7f5adcd99640 4 set_mon_vals no callback set 2024-03-25T18:19:10.872 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.870+0000 7f5aceffd640 1 -- 172.21.15.97:0/1097947061 <== mon.1 v1:172.21.15.98:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2377622137 0 0) 0x7f5ad401a920 con 0x7f5ae01473a0 2024-03-25T18:19:10.872 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.870+0000 7f5ade59c640 10 monclient: get_auth_request con 0x7f5ac006e0b0 auth_method 0 2024-03-25T18:19:10.872 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.871+0000 7f5aceffd640 10 client.4348.objecter ms_handle_connect 0x7f5ac006e0b0 2024-03-25T18:19:10.873 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.872+0000 7f5aceffd640 1 -- 172.21.15.97:0/1097947061 <== mon.1 v1:172.21.15.98:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 430711918 0 0) 0x7f5ad4054640 con 0x7f5ae01473a0 2024-03-25T18:19:10.873 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.872+0000 7f5aceffd640 10 client.4348.objecter ms_dispatch 0x7f5ae0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:19:10.873 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.872+0000 7f5aceffd640 3 client.4348.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:19:10.873 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.872+0000 7f5aceffd640 3 client.4348.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:19:10.873 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.872+0000 7f5aceffd640 20 client.4348.objecter dump_active .. 0 homeless 2024-03-25T18:19:10.875 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.874+0000 7f5aceffd640 1 -- 172.21.15.97:0/1097947061 <== mon.1 v1:172.21.15.98:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5ad404ff80 con 0x7f5ae01473a0 2024-03-25T18:19:10.875 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.874+0000 7f5aceffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:10.876 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:10.874+0000 7f5aceffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:19:11.108 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:11.106+0000 7f5ae4d54640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:11.108 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:11.106+0000 7f5ae4d54640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:11.108 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:11.107+0000 7f5ae4d54640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:19:11.108 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:11.107+0000 7f5ae4d54640 1 -- 172.21.15.97:0/1097947061 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5ae00627b0 con 0x7f5ae01473a0 2024-03-25T18:19:11.125 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:11.124+0000 7f5aceffd640 1 -- 172.21.15.97:0/1097947061 <== mon.1 v1:172.21.15.98:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2169427412) 0x7f5ad4052080 con 0x7f5ae01473a0 2024-03-25T18:19:11.125 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:11.124+0000 7f5aceffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:11.125 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:11.124+0000 7f5aceffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:19:11.125 INFO:teuthology.misc.health.smithi097.stdout: 2024-03-25T18:19:11.125 INFO:teuthology.misc.health.smithi097.stdout:{"epoch":10,"fsid":"4c3d472c-79b8-486b-9e3d-2738d4306754","created":"2024-03-25T18:16:14.737309+0000","modified":"2024-03-25T18:16:20.357492+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:20.357492+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cb5152b9-262f-45c3-94b8-18d0e81df558","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6daf8793-0838-473d-a7d6-9a0d2894c75d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"136f1493-3b9c-4f43-a2e7-641482d69dae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"397d538c-4f64-4daf-9e8b-662d09f91327","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4f1df08-663f-4bda-93f8-3878ab7c24b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c4b18d70-a3ce-4b15-9b70-eae5525bd761","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8dd4033a-de0e-4f8a-96dc-59f3c1f825b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0077722e-ffa6-4a3d-9891-929d9869b286","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:19:11.128 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:11.126+0000 7f5ae4d54640 10 librados: watch_flush enter 2024-03-25T18:19:11.128 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:11.126+0000 7f5ae4d54640 10 librados: watch_flush exit 2024-03-25T18:19:11.128 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:11.126+0000 7f5ae4d54640 20 client.4348.objecter shutdown clearing up homeless session... 2024-03-25T18:19:11.128 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:11.126+0000 7f5ae4d54640 10 client.4348.objecter successfully canceled tick 2024-03-25T18:19:11.128 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:11.126+0000 7f5ae4d54640 1 -- 172.21.15.97:0/1097947061 >> v1:172.21.15.98:6800/800387573 conn(0x7f5ac006e0b0 legacy=0x7f5ac0070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:11.128 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:11.126+0000 7f5ae4d54640 10 monclient: shutdown 2024-03-25T18:19:11.129 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:11.126+0000 7f5ae4d54640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:11.129 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:11.126+0000 7f5ae4d54640 1 -- 172.21.15.97:0/1097947061 >> v1:172.21.15.98:6789/0 conn(0x7f5ae01473a0 legacy=0x7f5ae0070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:11.129 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:11.126+0000 7f5ae4d54640 1 -- 172.21.15.97:0/1097947061 shutdown_connections 2024-03-25T18:19:11.129 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:11.126+0000 7f5ae4d54640 1 -- 172.21.15.97:0/1097947061 >> 172.21.15.97:0/1097947061 conn(0x7f5ae00b7ea0 msgr2=0x7f5ae01416c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:11.129 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:11.126+0000 7f5ae4d54640 1 -- 172.21.15.97:0/1097947061 shutdown_connections 2024-03-25T18:19:11.129 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:11.126+0000 7f5ae4d54640 1 -- 172.21.15.97:0/1097947061 wait complete. 2024-03-25T18:19:11.129 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:11.126+0000 7f5ae4d54640 1 librados: shutdown 2024-03-25T18:19:11.141 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:19:12.291 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:03:14.329 34931== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:12.343 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:03:14.382 34931== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:12.363 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:03:14.402 34931== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:12.394 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:03:14.437 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:12.399 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:03:14.443 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:12.404 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:03:14.443 34935== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:12.418 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:03:14.457 34935== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:12.432 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:03:14.476 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:12.452 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:03:14.491 34935== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:17.947 DEBUG:teuthology.orchestra.run.smithi097:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:19:18.060 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.059+0000 7f54cd02e640 10 monclient: get_monmap_and_config 2024-03-25T18:19:18.060 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.059+0000 7f54cd02e640 10 monclient: build_initial_monmap 2024-03-25T18:19:18.060 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.059+0000 7f54cd02e640 10 monclient: monmap: 2024-03-25T18:19:18.060 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:19:18.060 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:19:18.061 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:19:18.060477+0000 2024-03-25T18:19:18.061 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:19:18.060477+0000 2024-03-25T18:19:18.061 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:18.061 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:19:18.061 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:19:18.061 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T18:19:18.061 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-b 2024-03-25T18:19:18.061 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:19:18.063 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.059+0000 7f54cd02e640 1 Processor -- start 2024-03-25T18:19:18.063 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.059+0000 7f54cd02e640 1 -- start start 2024-03-25T18:19:18.063 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.059+0000 7f54cd02e640 10 monclient: init 2024-03-25T18:19:18.063 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.060+0000 7f54cd02e640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:18.063 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.060+0000 7f54cd02e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:19:18.063 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.060+0000 7f54cd02e640 10 monclient(hunting): picked mon.noname-b con 0x7f54c80a98f0 addr v1:172.21.15.97:6790/0 2024-03-25T18:19:18.063 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.060+0000 7f54cd02e640 10 monclient(hunting): picked mon.noname-a con 0x7f54c813f710 addr v1:172.21.15.97:6789/0 2024-03-25T18:19:18.063 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.060+0000 7f54cd02e640 10 monclient(hunting): picked mon.noname-c con 0x7f54c81430c0 addr v1:172.21.15.98:6789/0 2024-03-25T18:19:18.063 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.060+0000 7f54cd02e640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54c8142520 con 0x7f54c813f710 2024-03-25T18:19:18.063 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.060+0000 7f54cd02e640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54c80b28a0 con 0x7f54c81430c0 2024-03-25T18:19:18.064 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.060+0000 7f54cd02e640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54c80b2a30 con 0x7f54c80a98f0 2024-03-25T18:19:18.064 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.060+0000 7f54cd02e640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:18.064 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.060+0000 7f54cd02e640 10 monclient(hunting): authenticate will time out at 7770.336551s 2024-03-25T18:19:18.073 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.072+0000 7f54c6d9d640 1 --1- >> v1:172.21.15.97:6790/0 conn(0x7f54c80a98f0 0x7f54c80a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6790/0 says I am v1:172.21.15.97:57628/0 (socket says 172.21.15.97:57628) 2024-03-25T18:19:18.074 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.072+0000 7f54c6d9d640 1 -- 172.21.15.97:0/3840637870 learned_addr learned my addr 172.21.15.97:0/3840637870 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:19:18.094 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.093+0000 7f54c659c640 1 -- 172.21.15.97:0/3840637870 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2026956180 0 0) 0x7f54c80b2a30 con 0x7f54c80a98f0 2024-03-25T18:19:18.095 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.093+0000 7f54c659c640 10 monclient(hunting): _negotiate 2024-03-25T18:19:18.095 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.093+0000 7f54c659c640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:18.095 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.093+0000 7f54c659c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:18.095 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.093+0000 7f54c659c640 10 monclient(hunting): my global_id is 4346 2024-03-25T18:19:18.095 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.093+0000 7f54c659c640 1 -- 172.21.15.97:0/3840637870 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54b4003520 con 0x7f54c80a98f0 2024-03-25T18:19:18.097 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.095+0000 7f54c659c640 1 -- 172.21.15.97:0/3840637870 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1726549720 0 0) 0x7f54c80b28a0 con 0x7f54c81430c0 2024-03-25T18:19:18.097 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.095+0000 7f54c659c640 10 monclient(hunting): _negotiate 2024-03-25T18:19:18.097 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.095+0000 7f54c659c640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:18.097 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.095+0000 7f54c659c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:18.097 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.095+0000 7f54c659c640 10 monclient(hunting): my global_id is 4351 2024-03-25T18:19:18.097 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.095+0000 7f54c659c640 1 -- 172.21.15.97:0/3840637870 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54c80b2a30 con 0x7f54c81430c0 2024-03-25T18:19:18.110 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.109+0000 7f54c659c640 1 -- 172.21.15.97:0/3840637870 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3537407799 0 0) 0x7f54c80b2a30 con 0x7f54c81430c0 2024-03-25T18:19:18.111 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.109+0000 7f54c659c640 1 -- 172.21.15.97:0/3840637870 >> v1:172.21.15.97:6790/0 conn(0x7f54c80a98f0 legacy=0x7f54c80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:18.111 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.109+0000 7f54c659c640 1 -- 172.21.15.97:0/3840637870 >> v1:172.21.15.97:6789/0 conn(0x7f54c813f710 legacy=0x7f54c81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:18.111 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.109+0000 7f54c659c640 10 monclient: _finish_hunting 0 2024-03-25T18:19:18.111 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.109+0000 7f54c659c640 1 monclient: found mon.noname-c 2024-03-25T18:19:18.111 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.109+0000 7f54c659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:18.111 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.109+0000 7f54c659c640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.98:6789/0 2024-03-25T18:19:18.111 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.109+0000 7f54c659c640 1 -- 172.21.15.97:0/3840637870 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54c80b44c0 con 0x7f54c81430c0 2024-03-25T18:19:18.111 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.109+0000 7f54c659c640 10 monclient: _finish_auth 0 2024-03-25T18:19:18.111 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.109+0000 7f54c659c640 10 monclient: _check_auth_tickets 2024-03-25T18:19:18.111 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.109+0000 7f54c659c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:18.111 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.109+0000 7f54c659c640 1 -- 172.21.15.97:0/3840637870 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f54c00019f0 con 0x7f54c81430c0 2024-03-25T18:19:18.111 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.109+0000 7f54c659c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:18.111 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.109+0000 7f54c659c640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:19:18.112 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.109+0000 7f54c659c640 10 monclient: dump: 2024-03-25T18:19:18.112 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:19:18.112 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:19:18.112 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:19:18.112 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:19:18.112 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:18.112 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:19:18.112 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:19:18.112 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:19:18.112 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:19:18.112 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:19:18.112 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.109+0000 7f54cd02e640 5 monclient: authenticate success, global_id 4351 2024-03-25T18:19:18.112 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.109+0000 7f54cd02e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:18.145 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.144+0000 7f54c659c640 1 -- 172.21.15.97:0/3840637870 <== mon.1 v1:172.21.15.98:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f54c0001b50 con 0x7f54c81430c0 2024-03-25T18:19:18.145 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.144+0000 7f54c659c640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:18.146 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.144+0000 7f54c659c640 1 -- 172.21.15.97:0/3840637870 <== mon.1 v1:172.21.15.98:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f54c0001fa0 con 0x7f54c81430c0 2024-03-25T18:19:18.146 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.144+0000 7f54c659c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:18.146 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.144+0000 7f54c659c640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:19:18.146 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.144+0000 7f54c659c640 10 monclient: dump: 2024-03-25T18:19:18.146 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:19:18.146 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:19:18.146 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:19:18.146 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:19:18.146 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:18.147 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:19:18.147 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:19:18.147 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:19:18.147 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:19:18.147 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:19:18.147 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.144+0000 7f54cd02e640 10 monclient: get_monmap_and_config success 2024-03-25T18:19:18.147 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.144+0000 7f54cd02e640 4 set_mon_vals no callback set 2024-03-25T18:19:18.149 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.145+0000 7f54cd02e640 10 monclient: shutdown 2024-03-25T18:19:18.149 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.145+0000 7f54cd02e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:18.150 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.145+0000 7f54cd02e640 1 -- 172.21.15.97:0/3840637870 >> v1:172.21.15.98:6789/0 conn(0x7f54c81430c0 legacy=0x7f54c814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:18.150 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.145+0000 7f54cd02e640 1 -- 172.21.15.97:0/3840637870 shutdown_connections 2024-03-25T18:19:18.150 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.145+0000 7f54cd02e640 1 -- 172.21.15.97:0/3840637870 >> 172.21.15.97:0/3840637870 conn(0x7f54c80b7ea0 msgr2=0x7f54c80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:18.150 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.145+0000 7f54cd02e640 1 -- 172.21.15.97:0/3840637870 shutdown_connections 2024-03-25T18:19:18.150 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.145+0000 7f54cd02e640 1 -- 172.21.15.97:0/3840637870 wait complete. 2024-03-25T18:19:18.150 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.146+0000 7f54cd02e640 10 monclient: build_initial_monmap 2024-03-25T18:19:18.150 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.146+0000 7f54cd02e640 10 monclient: monmap: 2024-03-25T18:19:18.150 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:19:18.150 INFO:teuthology.misc.health.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:19:18.150 INFO:teuthology.misc.health.smithi097.stderr:last_changed 0.000000 2024-03-25T18:19:18.150 INFO:teuthology.misc.health.smithi097.stderr:created 0.000000 2024-03-25T18:19:18.151 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:18.151 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 1 2024-03-25T18:19:18.151 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:19:18.151 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T18:19:18.151 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-c 2024-03-25T18:19:18.151 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:19:18.151 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.146+0000 7f54cd02e640 1 librados: starting msgr at 2024-03-25T18:19:18.151 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.146+0000 7f54cd02e640 1 librados: starting objecter 2024-03-25T18:19:18.151 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.147+0000 7f54cd02e640 1 Processor -- start 2024-03-25T18:19:18.151 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.147+0000 7f54cd02e640 1 -- start start 2024-03-25T18:19:18.151 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.147+0000 7f54cd02e640 1 librados: setting wanted keys 2024-03-25T18:19:18.151 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.147+0000 7f54cd02e640 1 librados: calling monclient init 2024-03-25T18:19:18.151 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.147+0000 7f54cd02e640 10 monclient: init 2024-03-25T18:19:18.152 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.147+0000 7f54cd02e640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:18.152 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.147+0000 7f54cd02e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:19:18.152 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.147+0000 7f54cd02e640 10 monclient(hunting): picked mon.noname-b con 0x7f54c80a98f0 addr v1:172.21.15.98:6789/0 2024-03-25T18:19:18.152 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.147+0000 7f54cd02e640 10 monclient(hunting): picked mon.noname-c con 0x7f54c813f710 addr v1:172.21.15.97:6790/0 2024-03-25T18:19:18.152 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.147+0000 7f54cd02e640 10 monclient(hunting): picked mon.noname-a con 0x7f54c81430c0 addr v1:172.21.15.97:6789/0 2024-03-25T18:19:18.152 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.147+0000 7f54cd02e640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54c8138ad0 con 0x7f54c81430c0 2024-03-25T18:19:18.152 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.148+0000 7f54cd02e640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54c81d7dc0 con 0x7f54c80a98f0 2024-03-25T18:19:18.152 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.148+0000 7f54cd02e640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54c81d7f50 con 0x7f54c813f710 2024-03-25T18:19:18.152 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.148+0000 7f54cd02e640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:18.152 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.148+0000 7f54cd02e640 10 monclient(hunting): authenticate will time out at 7770.423920s 2024-03-25T18:19:18.153 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.150+0000 7f54be59c640 1 --1- >> v1:172.21.15.97:6790/0 conn(0x7f54c813f710 0x7f54c81379b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.97:6790/0 says I am v1:172.21.15.97:57684/0 (socket says 172.21.15.97:57684) 2024-03-25T18:19:18.153 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.150+0000 7f54be59c640 1 -- 172.21.15.97:0/4046154680 learned_addr learned my addr 172.21.15.97:0/4046154680 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:19:18.153 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.152+0000 7f54bffff640 10 client.?.objecter ms_handle_connect 0x7f54c813f710 2024-03-25T18:19:18.153 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.152+0000 7f54bffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:18.161 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.160+0000 7f54bffff640 1 -- 172.21.15.97:0/4046154680 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1044091791 0 0) 0x7f54c81d7f50 con 0x7f54c813f710 2024-03-25T18:19:18.162 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.160+0000 7f54bffff640 10 monclient(hunting): _negotiate 2024-03-25T18:19:18.162 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.160+0000 7f54bffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:18.162 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.160+0000 7f54bffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:18.162 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.160+0000 7f54bffff640 10 monclient(hunting): my global_id is 4349 2024-03-25T18:19:18.162 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.160+0000 7f54bffff640 1 -- 172.21.15.97:0/4046154680 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54a8004eb0 con 0x7f54c813f710 2024-03-25T18:19:18.164 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.163+0000 7f54bffff640 10 client.?.objecter ms_handle_connect 0x7f54c80a98f0 2024-03-25T18:19:18.164 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.163+0000 7f54bffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:18.168 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.167+0000 7f54bffff640 10 client.?.objecter ms_handle_connect 0x7f54c81430c0 2024-03-25T18:19:18.168 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.167+0000 7f54bffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:18.172 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.171+0000 7f54bffff640 1 -- 172.21.15.97:0/4046154680 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4155828606 0 0) 0x7f54c8138ad0 con 0x7f54c81430c0 2024-03-25T18:19:18.173 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.171+0000 7f54bffff640 10 monclient(hunting): _negotiate 2024-03-25T18:19:18.173 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.171+0000 7f54bffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:18.173 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.171+0000 7f54bffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:18.173 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.171+0000 7f54bffff640 10 monclient(hunting): my global_id is 4356 2024-03-25T18:19:18.173 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.171+0000 7f54bffff640 1 -- 172.21.15.97:0/4046154680 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54c81d7f50 con 0x7f54c81430c0 2024-03-25T18:19:18.177 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.176+0000 7f54bffff640 1 -- 172.21.15.97:0/4046154680 <== mon.2 v1:172.21.15.97:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1789923669 0 0) 0x7f54a8004eb0 con 0x7f54c813f710 2024-03-25T18:19:18.177 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.176+0000 7f54bffff640 1 -- 172.21.15.97:0/4046154680 --> v1:172.21.15.97:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f54c8138ad0 con 0x7f54c813f710 2024-03-25T18:19:18.177 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.176+0000 7f54bffff640 1 -- 172.21.15.97:0/4046154680 <== mon.2 v1:172.21.15.97:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f54b00019d0 con 0x7f54c813f710 2024-03-25T18:19:18.177 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.176+0000 7f54bffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:18.177 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.176+0000 7f54bffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:19:18.177 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.176+0000 7f54bffff640 10 monclient(hunting): dump: 2024-03-25T18:19:18.177 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:19:18.177 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:19:18.178 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:19:18.178 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:19:18.178 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:18.178 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:19:18.178 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:19:18.178 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:19:18.178 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:19:18.178 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:19:18.178 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.176+0000 7f54bffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:19:18.182 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.181+0000 7f54bffff640 1 -- 172.21.15.97:0/4046154680 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 565860865 0 0) 0x7f54c81d7dc0 con 0x7f54c80a98f0 2024-03-25T18:19:18.182 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.181+0000 7f54bffff640 10 monclient(hunting): _negotiate 2024-03-25T18:19:18.182 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.181+0000 7f54bffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:18.182 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.181+0000 7f54bffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:18.182 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.181+0000 7f54bffff640 10 monclient(hunting): my global_id is 4354 2024-03-25T18:19:18.183 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.181+0000 7f54bffff640 1 -- 172.21.15.97:0/4046154680 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54a8004eb0 con 0x7f54c80a98f0 2024-03-25T18:19:18.185 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.184+0000 7f54bffff640 1 -- 172.21.15.97:0/4046154680 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2266028587 0 0) 0x7f54c81d7f50 con 0x7f54c81430c0 2024-03-25T18:19:18.185 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.184+0000 7f54bffff640 1 -- 172.21.15.97:0/4046154680 --> v1:172.21.15.97:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f54c81d7dc0 con 0x7f54c81430c0 2024-03-25T18:19:18.185 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.184+0000 7f54bffff640 1 -- 172.21.15.97:0/4046154680 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f54c0001e60 con 0x7f54c81430c0 2024-03-25T18:19:18.186 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.184+0000 7f54bffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:18.186 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.184+0000 7f54bffff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:19:18.186 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.184+0000 7f54bffff640 10 monclient(hunting): dump: 2024-03-25T18:19:18.186 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:19:18.186 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:19:18.186 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:19:18.186 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:19:18.186 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:18.186 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:19:18.186 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:19:18.186 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:19:18.186 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:19:18.187 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:19:18.194 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.192+0000 7f54bffff640 1 -- 172.21.15.97:0/4046154680 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2747372377 0 0) 0x7f54a8004eb0 con 0x7f54c80a98f0 2024-03-25T18:19:18.194 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.192+0000 7f54bffff640 1 -- 172.21.15.97:0/4046154680 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f54c81d7f50 con 0x7f54c80a98f0 2024-03-25T18:19:18.194 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.192+0000 7f54bffff640 1 -- 172.21.15.97:0/4046154680 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f54b8001950 con 0x7f54c80a98f0 2024-03-25T18:19:18.194 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.192+0000 7f54bffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:18.194 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.192+0000 7f54bffff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:19:18.194 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.192+0000 7f54bffff640 10 monclient(hunting): dump: 2024-03-25T18:19:18.194 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:19:18.195 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:19:18.195 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:19:18.195 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:19:18.195 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:18.195 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:19:18.195 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:19:18.195 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:19:18.195 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:19:18.195 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:19:18.204 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.202+0000 7f54bffff640 1 -- 172.21.15.97:0/4046154680 <== mon.1 v1:172.21.15.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2729400961 0 0) 0x7f54c81d7f50 con 0x7f54c80a98f0 2024-03-25T18:19:18.204 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.202+0000 7f54bffff640 1 -- 172.21.15.97:0/4046154680 >> v1:172.21.15.97:6790/0 conn(0x7f54c813f710 legacy=0x7f54c81379b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:18.204 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.202+0000 7f54bffff640 1 -- 172.21.15.97:0/4046154680 >> v1:172.21.15.97:6789/0 conn(0x7f54c81430c0 legacy=0x7f54c81d76b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:18.204 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.202+0000 7f54bffff640 10 monclient: _finish_hunting 0 2024-03-25T18:19:18.204 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.202+0000 7f54bffff640 1 monclient: found mon.b 2024-03-25T18:19:18.204 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.202+0000 7f54bffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:18.204 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.202+0000 7f54bffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:19:18.204 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.202+0000 7f54bffff640 1 -- 172.21.15.97:0/4046154680 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54c81d80e0 con 0x7f54c80a98f0 2024-03-25T18:19:18.204 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.202+0000 7f54bffff640 10 monclient: _finish_auth 0 2024-03-25T18:19:18.205 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.202+0000 7f54bffff640 10 monclient: _check_auth_tickets 2024-03-25T18:19:18.205 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.202+0000 7f54bffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:18.205 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.202+0000 7f54cd02e640 5 monclient: authenticate success, global_id 4354 2024-03-25T18:19:18.205 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.202+0000 7f54cd02e640 10 monclient: _renew_subs 2024-03-25T18:19:18.205 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.202+0000 7f54cd02e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:19:18.205 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.202+0000 7f54cd02e640 1 -- 172.21.15.97:0/4046154680 --> v1:172.21.15.98:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54c81d8390 con 0x7f54c80a98f0 2024-03-25T18:19:18.205 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.202+0000 7f54cd02e640 10 client.4354.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:19:18.205 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.202+0000 7f54cd02e640 10 monclient: _renew_subs 2024-03-25T18:19:18.205 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.202+0000 7f54cd02e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:19:18.205 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.202+0000 7f54cd02e640 1 -- 172.21.15.97:0/4046154680 --> v1:172.21.15.98:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f54c81e31a0 con 0x7f54c80a98f0 2024-03-25T18:19:18.205 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.202+0000 7f54cd02e640 1 librados: init done 2024-03-25T18:19:18.206 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.203+0000 7f54cd02e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:18.206 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.203+0000 7f54cd02e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:18.206 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.203+0000 7f54cd02e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:19:18.206 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.203+0000 7f54cd02e640 1 -- 172.21.15.97:0/4046154680 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54c80627b0 con 0x7f54c80a98f0 2024-03-25T18:19:18.480 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.478+0000 7f54bffff640 1 -- 172.21.15.97:0/4046154680 <== mon.1 v1:172.21.15.98:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f54b80047e0 con 0x7f54c80a98f0 2024-03-25T18:19:18.480 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.478+0000 7f54bffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:18.480 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.478+0000 7f54bffff640 1 -- 172.21.15.97:0/4046154680 <== mon.1 v1:172.21.15.98:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f54b8001d80 con 0x7f54c80a98f0 2024-03-25T18:19:18.480 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.478+0000 7f54bffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:18.481 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.478+0000 7f54bffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:19:18.481 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.478+0000 7f54bffff640 10 monclient: dump: 2024-03-25T18:19:18.481 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:19:18.481 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:19:18.481 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:19:18.481 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:19:18.481 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:18.481 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:19:18.481 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:19:18.481 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:19:18.482 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:19:18.482 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:19:18.482 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.478+0000 7f54c5d9b640 4 set_mon_vals no callback set 2024-03-25T18:19:18.482 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.479+0000 7f54bffff640 1 -- 172.21.15.97:0/4046154680 <== mon.1 v1:172.21.15.98:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2377622137 0 0) 0x7f54b801aa20 con 0x7f54c80a98f0 2024-03-25T18:19:18.482 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.480+0000 7f54be59c640 10 monclient: get_auth_request con 0x7f54a806e0e0 auth_method 0 2024-03-25T18:19:18.483 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.480+0000 7f54bffff640 10 client.4354.objecter ms_handle_connect 0x7f54a806e0e0 2024-03-25T18:19:18.489 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.488+0000 7f54bffff640 1 -- 172.21.15.97:0/4046154680 <== mon.1 v1:172.21.15.98:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 430711918 0 0) 0x7f54b8054720 con 0x7f54c80a98f0 2024-03-25T18:19:18.490 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.488+0000 7f54bffff640 10 client.4354.objecter ms_dispatch 0x7f54c8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:19:18.490 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.488+0000 7f54bffff640 3 client.4354.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:19:18.490 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.488+0000 7f54bffff640 3 client.4354.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:19:18.490 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.488+0000 7f54bffff640 20 client.4354.objecter dump_active .. 0 homeless 2024-03-25T18:19:18.490 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.489+0000 7f54bffff640 1 -- 172.21.15.97:0/4046154680 <== mon.1 v1:172.21.15.98:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f54b8050060 con 0x7f54c80a98f0 2024-03-25T18:19:18.490 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.489+0000 7f54bffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:18.490 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.489+0000 7f54bffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:19:18.757 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.755+0000 7f54cd02e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:18.757 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.755+0000 7f54cd02e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:18.757 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.755+0000 7f54cd02e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:19:18.757 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.755+0000 7f54cd02e640 1 -- 172.21.15.97:0/4046154680 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f54c8141a90 con 0x7f54c80a98f0 2024-03-25T18:19:18.779 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.777+0000 7f54bffff640 1 -- 172.21.15.97:0/4046154680 <== mon.1 v1:172.21.15.98:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2169427412) 0x7f54b8052160 con 0x7f54c80a98f0 2024-03-25T18:19:18.779 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.778+0000 7f54bffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:18.779 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.778+0000 7f54bffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:19:18.779 INFO:teuthology.misc.health.smithi097.stdout: 2024-03-25T18:19:18.779 INFO:teuthology.misc.health.smithi097.stdout:{"epoch":10,"fsid":"4c3d472c-79b8-486b-9e3d-2738d4306754","created":"2024-03-25T18:16:14.737309+0000","modified":"2024-03-25T18:16:20.357492+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:20.357492+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cb5152b9-262f-45c3-94b8-18d0e81df558","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6daf8793-0838-473d-a7d6-9a0d2894c75d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"136f1493-3b9c-4f43-a2e7-641482d69dae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"397d538c-4f64-4daf-9e8b-662d09f91327","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4f1df08-663f-4bda-93f8-3878ab7c24b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c4b18d70-a3ce-4b15-9b70-eae5525bd761","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8dd4033a-de0e-4f8a-96dc-59f3c1f825b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0077722e-ffa6-4a3d-9891-929d9869b286","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:19:18.782 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.780+0000 7f54cd02e640 10 librados: watch_flush enter 2024-03-25T18:19:18.783 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.780+0000 7f54cd02e640 10 librados: watch_flush exit 2024-03-25T18:19:18.783 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.780+0000 7f54cd02e640 20 client.4354.objecter shutdown clearing up homeless session... 2024-03-25T18:19:18.783 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.780+0000 7f54cd02e640 10 client.4354.objecter successfully canceled tick 2024-03-25T18:19:18.783 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.780+0000 7f54cd02e640 1 -- 172.21.15.97:0/4046154680 >> v1:172.21.15.98:6800/800387573 conn(0x7f54a806e0e0 legacy=0x7f54a80705a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:18.783 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.780+0000 7f54cd02e640 10 monclient: shutdown 2024-03-25T18:19:18.783 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.780+0000 7f54cd02e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:18.783 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.780+0000 7f54cd02e640 1 -- 172.21.15.97:0/4046154680 >> v1:172.21.15.98:6789/0 conn(0x7f54c80a98f0 legacy=0x7f54c8138270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:18.783 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.780+0000 7f54cd02e640 1 -- 172.21.15.97:0/4046154680 shutdown_connections 2024-03-25T18:19:18.783 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.780+0000 7f54cd02e640 1 -- 172.21.15.97:0/4046154680 >> 172.21.15.97:0/4046154680 conn(0x7f54c80b7ea0 msgr2=0x7f54c8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:18.783 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.781+0000 7f54cd02e640 1 -- 172.21.15.97:0/4046154680 shutdown_connections 2024-03-25T18:19:18.783 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.781+0000 7f54cd02e640 1 -- 172.21.15.97:0/4046154680 wait complete. 2024-03-25T18:19:18.783 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:18.781+0000 7f54cd02e640 1 librados: shutdown 2024-03-25T18:19:18.793 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:19:25.600 DEBUG:teuthology.orchestra.run.smithi097:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:19:25.716 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:25.715+0000 7fe7fd5c4640 10 monclient: get_monmap_and_config 2024-03-25T18:19:25.716 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:25.715+0000 7fe7fd5c4640 10 monclient: build_initial_monmap 2024-03-25T18:19:25.716 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:25.715+0000 7fe7fd5c4640 10 monclient: monmap: 2024-03-25T18:19:25.716 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:19:25.717 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:19:25.717 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:19:25.716641+0000 2024-03-25T18:19:25.717 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:19:25.716641+0000 2024-03-25T18:19:25.717 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:25.717 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:19:25.717 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:19:25.717 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T18:19:25.717 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-b 2024-03-25T18:19:25.717 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:19:25.718 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:25.716+0000 7fe7fd5c4640 1 Processor -- start 2024-03-25T18:19:25.718 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:25.716+0000 7fe7fd5c4640 1 -- start start 2024-03-25T18:19:25.718 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:25.716+0000 7fe7fd5c4640 10 monclient: init 2024-03-25T18:19:25.718 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:25.717+0000 7fe7fd5c4640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:25.718 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:25.717+0000 7fe7fd5c4640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:19:25.718 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:25.717+0000 7fe7fd5c4640 10 monclient(hunting): picked mon.noname-a con 0x7fe7f80a98f0 addr v1:172.21.15.97:6789/0 2024-03-25T18:19:25.718 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:25.717+0000 7fe7fd5c4640 10 monclient(hunting): picked mon.noname-b con 0x7fe7f813f710 addr v1:172.21.15.97:6790/0 2024-03-25T18:19:25.719 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:25.717+0000 7fe7fd5c4640 10 monclient(hunting): picked mon.noname-c con 0x7fe7f81430c0 addr v1:172.21.15.98:6789/0 2024-03-25T18:19:25.719 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:25.717+0000 7fe7fd5c4640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7f8142520 con 0x7fe7f80a98f0 2024-03-25T18:19:25.719 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:25.717+0000 7fe7fd5c4640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7f80b28a0 con 0x7fe7f81430c0 2024-03-25T18:19:25.719 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:25.717+0000 7fe7fd5c4640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7f80b2a30 con 0x7fe7f813f710 2024-03-25T18:19:25.719 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:25.717+0000 7fe7fd5c4640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:25.719 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:25.717+0000 7fe7fd5c4640 10 monclient(hunting): authenticate will time out at 7777.993063s 2024-03-25T18:19:25.732 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:25.731+0000 7fe7f77fe640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fe7f81430c0 0x7fe7f814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.97:56056/0 (socket says 172.21.15.97:56056) 2024-03-25T18:19:25.732 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:25.731+0000 7fe7f77fe640 1 -- 172.21.15.97:0/1843512483 learned_addr learned my addr 172.21.15.97:0/1843512483 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:19:25.750 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:25.749+0000 7fe7f67fc640 1 -- 172.21.15.97:0/1843512483 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1591818991 0 0) 0x7fe7f80b28a0 con 0x7fe7f81430c0 2024-03-25T18:19:25.750 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:25.749+0000 7fe7f67fc640 10 monclient(hunting): _negotiate 2024-03-25T18:19:25.750 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:25.749+0000 7fe7f67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:25.750 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:25.749+0000 7fe7f67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:25.751 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:25.749+0000 7fe7f67fc640 10 monclient(hunting): my global_id is 4357 2024-03-25T18:19:25.751 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:25.749+0000 7fe7f67fc640 1 -- 172.21.15.97:0/1843512483 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7e4003520 con 0x7fe7f81430c0 2024-03-25T18:19:25.753 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:25.751+0000 7fe7f67fc640 1 -- 172.21.15.97:0/1843512483 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2063132586 0 0) 0x7fe7f8142520 con 0x7fe7f80a98f0 2024-03-25T18:19:25.753 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:25.751+0000 7fe7f67fc640 10 monclient(hunting): _negotiate 2024-03-25T18:19:25.753 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:25.751+0000 7fe7f67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:25.753 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:25.751+0000 7fe7f67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:25.753 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:25.751+0000 7fe7f67fc640 10 monclient(hunting): my global_id is 4359 2024-03-25T18:19:25.753 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:25.751+0000 7fe7f67fc640 1 -- 172.21.15.97:0/1843512483 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7f80b28a0 con 0x7fe7f80a98f0 2024-03-25T18:19:25.763 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:25.761+0000 7fe7f67fc640 1 -- 172.21.15.97:0/1843512483 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4030875162 0 0) 0x7fe7f80b28a0 con 0x7fe7f80a98f0 2024-03-25T18:19:25.763 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:25.761+0000 7fe7f67fc640 1 -- 172.21.15.97:0/1843512483 >> v1:172.21.15.97:6790/0 conn(0x7fe7f813f710 legacy=0x7fe7f81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:25.763 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:25.761+0000 7fe7f67fc640 1 -- 172.21.15.97:0/1843512483 >> v1:172.21.15.98:6789/0 conn(0x7fe7f81430c0 legacy=0x7fe7f814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:25.763 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:25.761+0000 7fe7f67fc640 10 monclient: _finish_hunting 0 2024-03-25T18:19:25.763 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:25.761+0000 7fe7f67fc640 1 monclient: found mon.noname-a 2024-03-25T18:19:25.763 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:25.761+0000 7fe7f67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:25.763 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:25.761+0000 7fe7f67fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.97:6789/0 2024-03-25T18:19:25.763 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:25.761+0000 7fe7f67fc640 1 -- 172.21.15.97:0/1843512483 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7f80b44c0 con 0x7fe7f80a98f0 2024-03-25T18:19:25.763 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:25.761+0000 7fe7f67fc640 10 monclient: _finish_auth 0 2024-03-25T18:19:25.763 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:25.761+0000 7fe7f67fc640 10 monclient: _check_auth_tickets 2024-03-25T18:19:25.763 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:25.761+0000 7fe7f67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:25.763 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:25.761+0000 7fe7f67fc640 1 -- 172.21.15.97:0/1843512483 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7fe7e8001670 con 0x7fe7f80a98f0 2024-03-25T18:19:25.763 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:25.761+0000 7fe7fd5c4640 5 monclient: authenticate success, global_id 4359 2024-03-25T18:19:25.764 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:25.761+0000 7fe7fd5c4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:25.764 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:25.761+0000 7fe7f67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:25.764 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:25.761+0000 7fe7f67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:19:25.764 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:25.761+0000 7fe7f67fc640 10 monclient: dump: 2024-03-25T18:19:25.764 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:19:25.764 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:19:25.764 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:19:25.764 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:19:25.764 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:25.764 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:19:25.764 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:19:25.764 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:19:25.764 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:19:25.764 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:19:25.764 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:25.761+0000 7fe7fd5c4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:25.806 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:25.805+0000 7fe7f67fc640 1 -- 172.21.15.97:0/1843512483 <== mon.0 v1:172.21.15.97:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe7e80017d0 con 0x7fe7f80a98f0 2024-03-25T18:19:25.806 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:25.805+0000 7fe7f67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:25.806 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:25.805+0000 7fe7f67fc640 1 -- 172.21.15.97:0/1843512483 <== mon.0 v1:172.21.15.97:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7fe7e8001b60 con 0x7fe7f80a98f0 2024-03-25T18:19:25.806 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:25.805+0000 7fe7fd5c4640 10 monclient: get_monmap_and_config success 2024-03-25T18:19:25.806 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:25.805+0000 7fe7fd5c4640 4 set_mon_vals no callback set 2024-03-25T18:19:25.806 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:25.805+0000 7fe7f67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:25.806 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:25.805+0000 7fe7f67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:19:25.806 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:25.805+0000 7fe7f67fc640 10 monclient: dump: 2024-03-25T18:19:25.806 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:19:25.806 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:19:25.807 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:19:25.807 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:19:25.807 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:25.807 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:19:25.807 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:19:25.807 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:19:25.807 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:19:25.807 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:19:25.808 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:25.805+0000 7fe7fd5c4640 10 monclient: shutdown 2024-03-25T18:19:25.808 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:25.805+0000 7fe7fd5c4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:25.808 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:25.805+0000 7fe7fd5c4640 1 -- 172.21.15.97:0/1843512483 >> v1:172.21.15.97:6789/0 conn(0x7fe7f80a98f0 legacy=0x7fe7f80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:25.808 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:25.806+0000 7fe7fd5c4640 1 -- 172.21.15.97:0/1843512483 shutdown_connections 2024-03-25T18:19:25.808 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:25.806+0000 7fe7fd5c4640 1 -- 172.21.15.97:0/1843512483 >> 172.21.15.97:0/1843512483 conn(0x7fe7f80b7ea0 msgr2=0x7fe7f80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:25.809 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:25.806+0000 7fe7fd5c4640 1 -- 172.21.15.97:0/1843512483 shutdown_connections 2024-03-25T18:19:25.809 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:25.806+0000 7fe7fd5c4640 1 -- 172.21.15.97:0/1843512483 wait complete. 2024-03-25T18:19:25.809 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:25.806+0000 7fe7fd5c4640 10 monclient: build_initial_monmap 2024-03-25T18:19:25.809 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:25.806+0000 7fe7fd5c4640 10 monclient: monmap: 2024-03-25T18:19:25.809 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:19:25.809 INFO:teuthology.misc.health.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:19:25.809 INFO:teuthology.misc.health.smithi097.stderr:last_changed 0.000000 2024-03-25T18:19:25.809 INFO:teuthology.misc.health.smithi097.stderr:created 0.000000 2024-03-25T18:19:25.809 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:25.809 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 1 2024-03-25T18:19:25.809 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:19:25.809 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T18:19:25.809 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-c 2024-03-25T18:19:25.809 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:19:25.809 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:25.807+0000 7fe7fd5c4640 1 librados: starting msgr at 2024-03-25T18:19:25.810 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:25.807+0000 7fe7fd5c4640 1 librados: starting objecter 2024-03-25T18:19:25.810 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:25.807+0000 7fe7fd5c4640 1 Processor -- start 2024-03-25T18:19:25.810 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:25.807+0000 7fe7fd5c4640 1 -- start start 2024-03-25T18:19:25.810 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:25.807+0000 7fe7fd5c4640 1 librados: setting wanted keys 2024-03-25T18:19:25.810 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:25.807+0000 7fe7fd5c4640 1 librados: calling monclient init 2024-03-25T18:19:25.810 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:25.807+0000 7fe7fd5c4640 10 monclient: init 2024-03-25T18:19:25.811 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:25.808+0000 7fe7fd5c4640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:25.811 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:25.808+0000 7fe7fd5c4640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:19:25.811 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:25.808+0000 7fe7fd5c4640 10 monclient(hunting): picked mon.noname-a con 0x7fe7f80a98f0 addr v1:172.21.15.97:6789/0 2024-03-25T18:19:25.811 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:25.808+0000 7fe7fd5c4640 10 monclient(hunting): picked mon.noname-c con 0x7fe7f813f710 addr v1:172.21.15.97:6790/0 2024-03-25T18:19:25.811 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:25.808+0000 7fe7fd5c4640 10 monclient(hunting): picked mon.noname-b con 0x7fe7f81430c0 addr v1:172.21.15.98:6789/0 2024-03-25T18:19:25.811 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:25.808+0000 7fe7fd5c4640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7f813bf90 con 0x7fe7f80a98f0 2024-03-25T18:19:25.811 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:25.808+0000 7fe7fd5c4640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7f813b060 con 0x7fe7f81430c0 2024-03-25T18:19:25.811 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:25.808+0000 7fe7fd5c4640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7f81dc070 con 0x7fe7f813f710 2024-03-25T18:19:25.811 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:25.808+0000 7fe7fd5c4640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:25.811 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:25.808+0000 7fe7fd5c4640 10 monclient(hunting): authenticate will time out at 7778.084245s 2024-03-25T18:19:25.825 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:25.824+0000 7fe7f6ffd640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fe7f80a98f0 0x7fe7f813c400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:39730/0 (socket says 172.21.15.97:39730) 2024-03-25T18:19:25.825 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:25.824+0000 7fe7f6ffd640 1 -- 172.21.15.97:0/3815615648 learned_addr learned my addr 172.21.15.97:0/3815615648 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:19:25.827 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:25.826+0000 7fe7ef7fe640 10 client.?.objecter ms_handle_connect 0x7fe7f80a98f0 2024-03-25T18:19:25.827 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:25.826+0000 7fe7ef7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:25.834 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:25.832+0000 7fe7ef7fe640 10 client.?.objecter ms_handle_connect 0x7fe7f81430c0 2024-03-25T18:19:25.834 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:25.832+0000 7fe7ef7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:25.841 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:25.840+0000 7fe7ef7fe640 10 client.?.objecter ms_handle_connect 0x7fe7f813f710 2024-03-25T18:19:25.841 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:25.840+0000 7fe7ef7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:25.844 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:25.843+0000 7fe7ef7fe640 1 -- 172.21.15.97:0/3815615648 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2354905269 0 0) 0x7fe7f813bf90 con 0x7fe7f80a98f0 2024-03-25T18:19:25.844 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:25.843+0000 7fe7ef7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:19:25.844 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:25.843+0000 7fe7ef7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:25.844 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:25.843+0000 7fe7ef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:25.844 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:25.843+0000 7fe7ef7fe640 10 monclient(hunting): my global_id is 4362 2024-03-25T18:19:25.844 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:25.843+0000 7fe7ef7fe640 1 -- 172.21.15.97:0/3815615648 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7d8004eb0 con 0x7fe7f80a98f0 2024-03-25T18:19:25.845 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:25.843+0000 7fe7ef7fe640 1 -- 172.21.15.97:0/3815615648 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3504400929 0 0) 0x7fe7f813b060 con 0x7fe7f81430c0 2024-03-25T18:19:25.845 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:25.843+0000 7fe7ef7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:19:25.845 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:25.843+0000 7fe7ef7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:25.845 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:25.843+0000 7fe7ef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:25.845 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:25.843+0000 7fe7ef7fe640 10 monclient(hunting): my global_id is 4360 2024-03-25T18:19:25.845 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:25.843+0000 7fe7ef7fe640 1 -- 172.21.15.97:0/3815615648 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7f813bf90 con 0x7fe7f81430c0 2024-03-25T18:19:25.845 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:25.844+0000 7fe7ef7fe640 1 -- 172.21.15.97:0/3815615648 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 329091861 0 0) 0x7fe7f81dc070 con 0x7fe7f813f710 2024-03-25T18:19:25.845 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:25.844+0000 7fe7ef7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:19:25.845 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:25.844+0000 7fe7ef7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:25.845 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:25.844+0000 7fe7ef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:25.845 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:25.844+0000 7fe7ef7fe640 10 monclient(hunting): my global_id is 4355 2024-03-25T18:19:25.846 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:25.844+0000 7fe7ef7fe640 1 -- 172.21.15.97:0/3815615648 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7f813b060 con 0x7fe7f813f710 2024-03-25T18:19:25.855 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:25.854+0000 7fe7ef7fe640 1 -- 172.21.15.97:0/3815615648 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2648743518 0 0) 0x7fe7f813bf90 con 0x7fe7f81430c0 2024-03-25T18:19:25.856 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:25.854+0000 7fe7ef7fe640 1 -- 172.21.15.97:0/3815615648 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe7f81dc070 con 0x7fe7f81430c0 2024-03-25T18:19:25.856 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:25.854+0000 7fe7ef7fe640 1 -- 172.21.15.97:0/3815615648 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7fe7f0001a70 con 0x7fe7f81430c0 2024-03-25T18:19:25.856 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:25.854+0000 7fe7ef7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:25.856 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:25.854+0000 7fe7ef7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:19:25.856 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:25.854+0000 7fe7ef7fe640 10 monclient(hunting): dump: 2024-03-25T18:19:25.856 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:19:25.856 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:19:25.856 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:19:25.856 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:19:25.856 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:25.856 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:19:25.856 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:19:25.856 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:19:25.856 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:19:25.856 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:19:25.857 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:25.854+0000 7fe7ef7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:19:25.867 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:25.866+0000 7fe7ef7fe640 1 -- 172.21.15.97:0/3815615648 <== mon.1 v1:172.21.15.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1579497022 0 0) 0x7fe7f81dc070 con 0x7fe7f81430c0 2024-03-25T18:19:25.867 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:25.866+0000 7fe7ef7fe640 1 -- 172.21.15.97:0/3815615648 >> v1:172.21.15.97:6790/0 conn(0x7fe7f813f710 legacy=0x7fe7f81db250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:25.867 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:25.866+0000 7fe7ef7fe640 1 -- 172.21.15.97:0/3815615648 >> v1:172.21.15.97:6789/0 conn(0x7fe7f80a98f0 legacy=0x7fe7f813c400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:25.867 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:25.866+0000 7fe7ef7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:19:25.868 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:25.866+0000 7fe7ef7fe640 1 monclient: found mon.b 2024-03-25T18:19:25.868 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:25.866+0000 7fe7ef7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:25.868 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:25.866+0000 7fe7ef7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:19:25.868 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:25.866+0000 7fe7ef7fe640 1 -- 172.21.15.97:0/3815615648 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7f813b2d0 con 0x7fe7f81430c0 2024-03-25T18:19:25.868 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:25.866+0000 7fe7ef7fe640 10 monclient: _finish_auth 0 2024-03-25T18:19:25.868 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:25.866+0000 7fe7ef7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:19:25.868 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:25.866+0000 7fe7ef7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:25.868 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:25.866+0000 7fe7fd5c4640 5 monclient: authenticate success, global_id 4360 2024-03-25T18:19:25.868 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:25.866+0000 7fe7fd5c4640 10 monclient: _renew_subs 2024-03-25T18:19:25.868 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:25.866+0000 7fe7fd5c4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:19:25.868 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:25.866+0000 7fe7fd5c4640 1 -- 172.21.15.97:0/3815615648 --> v1:172.21.15.98:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7f813b5b0 con 0x7fe7f81430c0 2024-03-25T18:19:25.868 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:25.866+0000 7fe7fd5c4640 10 client.4360.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:19:25.868 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:25.866+0000 7fe7fd5c4640 10 monclient: _renew_subs 2024-03-25T18:19:25.868 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:25.866+0000 7fe7fd5c4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:19:25.868 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:25.866+0000 7fe7fd5c4640 1 -- 172.21.15.97:0/3815615648 --> v1:172.21.15.98:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7f80b7660 con 0x7fe7f81430c0 2024-03-25T18:19:25.868 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:25.866+0000 7fe7fd5c4640 1 librados: init done 2024-03-25T18:19:25.869 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:25.867+0000 7fe7fd5c4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:25.869 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:25.867+0000 7fe7fd5c4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:25.869 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:25.867+0000 7fe7fd5c4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:19:25.869 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:25.867+0000 7fe7fd5c4640 1 -- 172.21.15.97:0/3815615648 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7f80b3ef0 con 0x7fe7f81430c0 2024-03-25T18:19:26.147 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:26.146+0000 7fe7ef7fe640 1 -- 172.21.15.97:0/3815615648 <== mon.1 v1:172.21.15.98:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe7f0001ea0 con 0x7fe7f81430c0 2024-03-25T18:19:26.147 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:26.146+0000 7fe7ef7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:26.147 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:26.146+0000 7fe7ef7fe640 1 -- 172.21.15.97:0/3815615648 <== mon.1 v1:172.21.15.98:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7fe7f0004d00 con 0x7fe7f81430c0 2024-03-25T18:19:26.147 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:26.146+0000 7fe7ef7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:26.147 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:26.146+0000 7fe7ef7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:19:26.147 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:26.146+0000 7fe7f5ffb640 4 set_mon_vals no callback set 2024-03-25T18:19:26.148 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:26.146+0000 7fe7ef7fe640 10 monclient: dump: 2024-03-25T18:19:26.148 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:19:26.148 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:19:26.148 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:19:26.148 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:19:26.148 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:26.148 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:19:26.148 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:19:26.148 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:19:26.148 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:19:26.148 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:19:26.149 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:26.146+0000 7fe7ef7fe640 1 -- 172.21.15.97:0/3815615648 <== mon.1 v1:172.21.15.98:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2377622137 0 0) 0x7fe7f0004f20 con 0x7fe7f81430c0 2024-03-25T18:19:26.149 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:26.147+0000 7fe7f6ffd640 10 monclient: get_auth_request con 0x7fe7d806de20 auth_method 0 2024-03-25T18:19:26.149 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:26.147+0000 7fe7ef7fe640 10 client.4360.objecter ms_handle_connect 0x7fe7d806de20 2024-03-25T18:19:26.153 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:26.152+0000 7fe7ef7fe640 1 -- 172.21.15.97:0/3815615648 <== mon.1 v1:172.21.15.98:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 430711918 0 0) 0x7fe7f0054920 con 0x7fe7f81430c0 2024-03-25T18:19:26.153 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:26.152+0000 7fe7ef7fe640 10 client.4360.objecter ms_dispatch 0x7fe7f8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:19:26.154 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:26.152+0000 7fe7ef7fe640 3 client.4360.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:19:26.154 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:26.152+0000 7fe7ef7fe640 3 client.4360.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:19:26.154 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:26.152+0000 7fe7ef7fe640 20 client.4360.objecter dump_active .. 0 homeless 2024-03-25T18:19:26.154 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:26.153+0000 7fe7ef7fe640 1 -- 172.21.15.97:0/3815615648 <== mon.1 v1:172.21.15.98:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe7f0050430 con 0x7fe7f81430c0 2024-03-25T18:19:26.154 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:26.153+0000 7fe7ef7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:26.154 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:26.153+0000 7fe7ef7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:19:26.387 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:26.385+0000 7fe7fd5c4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:26.387 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:26.385+0000 7fe7fd5c4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:26.387 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:26.385+0000 7fe7fd5c4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:19:26.387 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:26.385+0000 7fe7fd5c4640 1 -- 172.21.15.97:0/3815615648 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe7f80627b0 con 0x7fe7f81430c0 2024-03-25T18:19:26.413 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:26.412+0000 7fe7ef7fe640 1 -- 172.21.15.97:0/3815615648 <== mon.1 v1:172.21.15.98:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2169427412) 0x7fe7f0052360 con 0x7fe7f81430c0 2024-03-25T18:19:26.413 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:26.412+0000 7fe7ef7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:26.413 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:26.412+0000 7fe7ef7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:19:26.413 INFO:teuthology.misc.health.smithi097.stdout: 2024-03-25T18:19:26.413 INFO:teuthology.misc.health.smithi097.stdout:{"epoch":10,"fsid":"4c3d472c-79b8-486b-9e3d-2738d4306754","created":"2024-03-25T18:16:14.737309+0000","modified":"2024-03-25T18:16:20.357492+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:20.357492+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cb5152b9-262f-45c3-94b8-18d0e81df558","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6daf8793-0838-473d-a7d6-9a0d2894c75d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"136f1493-3b9c-4f43-a2e7-641482d69dae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"397d538c-4f64-4daf-9e8b-662d09f91327","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4f1df08-663f-4bda-93f8-3878ab7c24b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c4b18d70-a3ce-4b15-9b70-eae5525bd761","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8dd4033a-de0e-4f8a-96dc-59f3c1f825b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0077722e-ffa6-4a3d-9891-929d9869b286","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:19:26.416 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:26.414+0000 7fe7fd5c4640 10 librados: watch_flush enter 2024-03-25T18:19:26.416 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:26.414+0000 7fe7fd5c4640 10 librados: watch_flush exit 2024-03-25T18:19:26.416 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:26.414+0000 7fe7fd5c4640 20 client.4360.objecter shutdown clearing up homeless session... 2024-03-25T18:19:26.416 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:26.414+0000 7fe7fd5c4640 10 client.4360.objecter successfully canceled tick 2024-03-25T18:19:26.416 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:26.414+0000 7fe7fd5c4640 1 -- 172.21.15.97:0/3815615648 >> v1:172.21.15.98:6800/800387573 conn(0x7fe7d806de20 legacy=0x7fe7d80702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:26.416 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:26.414+0000 7fe7fd5c4640 10 monclient: shutdown 2024-03-25T18:19:26.416 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:26.414+0000 7fe7fd5c4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:26.416 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:26.414+0000 7fe7fd5c4640 1 -- 172.21.15.97:0/3815615648 >> v1:172.21.15.98:6789/0 conn(0x7fe7f81430c0 legacy=0x7fe7f81db960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:26.416 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:26.414+0000 7fe7fd5c4640 1 -- 172.21.15.97:0/3815615648 shutdown_connections 2024-03-25T18:19:26.416 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:26.414+0000 7fe7fd5c4640 1 -- 172.21.15.97:0/3815615648 >> 172.21.15.97:0/3815615648 conn(0x7fe7f80b7ea0 msgr2=0x7fe7f8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:26.417 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:26.414+0000 7fe7fd5c4640 1 -- 172.21.15.97:0/3815615648 shutdown_connections 2024-03-25T18:19:26.417 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:26.414+0000 7fe7fd5c4640 1 -- 172.21.15.97:0/3815615648 wait complete. 2024-03-25T18:19:26.417 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:26.414+0000 7fe7fd5c4640 1 librados: shutdown 2024-03-25T18:19:26.427 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:19:32.398 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:03:34.437 34931== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:32.402 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:03:34.441 34931== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:32.450 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:03:34.489 34931== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:32.458 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:03:34.497 34935== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:32.462 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:03:34.505 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:32.466 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:03:34.505 34935== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:32.468 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:03:34.511 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:32.491 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:03:34.530 34935== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:32.499 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:03:34.542 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:33.234 DEBUG:teuthology.orchestra.run.smithi097:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:19:33.349 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.349+0000 7f65665a4640 10 monclient: get_monmap_and_config 2024-03-25T18:19:33.349 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.349+0000 7f65665a4640 10 monclient: build_initial_monmap 2024-03-25T18:19:33.350 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.349+0000 7f65665a4640 10 monclient: monmap: 2024-03-25T18:19:33.350 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:19:33.350 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:19:33.350 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:19:33.349647+0000 2024-03-25T18:19:33.350 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:19:33.349647+0000 2024-03-25T18:19:33.350 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:33.350 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:19:33.350 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:19:33.350 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T18:19:33.350 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-b 2024-03-25T18:19:33.350 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:19:33.351 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.350+0000 7f65665a4640 1 Processor -- start 2024-03-25T18:19:33.351 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.350+0000 7f65665a4640 1 -- start start 2024-03-25T18:19:33.351 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.350+0000 7f65665a4640 10 monclient: init 2024-03-25T18:19:33.352 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.350+0000 7f65665a4640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:33.352 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.350+0000 7f65665a4640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:19:33.352 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.350+0000 7f65665a4640 10 monclient(hunting): picked mon.noname-b con 0x7f65600a98f0 addr v1:172.21.15.97:6790/0 2024-03-25T18:19:33.352 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.350+0000 7f65665a4640 10 monclient(hunting): picked mon.noname-a con 0x7f656013f710 addr v1:172.21.15.97:6789/0 2024-03-25T18:19:33.352 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.351+0000 7f65665a4640 10 monclient(hunting): picked mon.noname-c con 0x7f65601430c0 addr v1:172.21.15.98:6789/0 2024-03-25T18:19:33.352 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.351+0000 7f65665a4640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6560142520 con 0x7f656013f710 2024-03-25T18:19:33.352 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.351+0000 7f65665a4640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65600b28a0 con 0x7f65601430c0 2024-03-25T18:19:33.352 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.351+0000 7f65665a4640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65600b2a30 con 0x7f65600a98f0 2024-03-25T18:19:33.352 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.351+0000 7f65665a4640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:33.352 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.351+0000 7f65665a4640 10 monclient(hunting): authenticate will time out at 7785.626020s 2024-03-25T18:19:33.367 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.367+0000 7f6564b41640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f65601430c0 0x7f656014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.97:47270/0 (socket says 172.21.15.97:47270) 2024-03-25T18:19:33.367 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.367+0000 7f6564b41640 1 -- 172.21.15.97:0/3192186956 learned_addr learned my addr 172.21.15.97:0/3192186956 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:19:33.378 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.378+0000 7f655effd640 1 -- 172.21.15.97:0/3192186956 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 762670629 0 0) 0x7f6560142520 con 0x7f656013f710 2024-03-25T18:19:33.379 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.378+0000 7f655effd640 10 monclient(hunting): _negotiate 2024-03-25T18:19:33.379 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.378+0000 7f655effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:33.379 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.378+0000 7f655effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:33.379 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.378+0000 7f655effd640 10 monclient(hunting): my global_id is 4365 2024-03-25T18:19:33.379 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.378+0000 7f655effd640 1 -- 172.21.15.97:0/3192186956 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6548003520 con 0x7f656013f710 2024-03-25T18:19:33.389 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.388+0000 7f655effd640 1 -- 172.21.15.97:0/3192186956 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3070611210 0 0) 0x7f6548003520 con 0x7f656013f710 2024-03-25T18:19:33.389 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.388+0000 7f655effd640 1 -- 172.21.15.97:0/3192186956 >> v1:172.21.15.97:6790/0 conn(0x7f65600a98f0 legacy=0x7f65600a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:33.389 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.388+0000 7f655effd640 1 -- 172.21.15.97:0/3192186956 >> v1:172.21.15.98:6789/0 conn(0x7f65601430c0 legacy=0x7f656014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:33.389 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.388+0000 7f655effd640 10 monclient: _finish_hunting 0 2024-03-25T18:19:33.389 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.388+0000 7f655effd640 1 monclient: found mon.noname-a 2024-03-25T18:19:33.389 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.388+0000 7f655effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:33.389 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.388+0000 7f655effd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.97:6789/0 2024-03-25T18:19:33.389 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.388+0000 7f655effd640 1 -- 172.21.15.97:0/3192186956 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65600b44c0 con 0x7f656013f710 2024-03-25T18:19:33.389 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.388+0000 7f655effd640 10 monclient: _finish_auth 0 2024-03-25T18:19:33.389 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.388+0000 7f655effd640 10 monclient: _check_auth_tickets 2024-03-25T18:19:33.390 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.388+0000 7f655effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:33.390 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.388+0000 7f655effd640 1 -- 172.21.15.97:0/3192186956 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f6554001690 con 0x7f656013f710 2024-03-25T18:19:33.390 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.388+0000 7f655effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:33.390 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.388+0000 7f655effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:19:33.390 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.388+0000 7f655effd640 10 monclient: dump: 2024-03-25T18:19:33.390 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:19:33.390 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:19:33.390 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:19:33.390 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:19:33.390 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:33.390 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:19:33.390 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:19:33.390 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:19:33.390 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:19:33.391 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:19:33.391 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.389+0000 7f65665a4640 5 monclient: authenticate success, global_id 4365 2024-03-25T18:19:33.391 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.389+0000 7f65665a4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:33.397 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.397+0000 7f655effd640 1 -- 172.21.15.97:0/3192186956 <== mon.0 v1:172.21.15.97:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f65540017f0 con 0x7f656013f710 2024-03-25T18:19:33.397 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.397+0000 7f655effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:33.397 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.397+0000 7f655effd640 1 -- 172.21.15.97:0/3192186956 <== mon.0 v1:172.21.15.97:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f6554001c00 con 0x7f656013f710 2024-03-25T18:19:33.397 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.397+0000 7f655effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:33.397 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.397+0000 7f655effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:19:33.397 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.397+0000 7f655effd640 10 monclient: dump: 2024-03-25T18:19:33.397 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:19:33.397 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:19:33.397 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:19:33.398 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:19:33.398 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:33.398 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:19:33.398 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:19:33.398 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:19:33.398 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:19:33.398 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:19:33.398 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.397+0000 7f65665a4640 10 monclient: get_monmap_and_config success 2024-03-25T18:19:33.398 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.397+0000 7f65665a4640 4 set_mon_vals no callback set 2024-03-25T18:19:33.399 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.398+0000 7f65665a4640 10 monclient: shutdown 2024-03-25T18:19:33.399 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.398+0000 7f65665a4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:33.399 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.398+0000 7f65665a4640 1 -- 172.21.15.97:0/3192186956 >> v1:172.21.15.97:6789/0 conn(0x7f656013f710 legacy=0x7f65601429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:33.399 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.398+0000 7f65665a4640 1 -- 172.21.15.97:0/3192186956 shutdown_connections 2024-03-25T18:19:33.399 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.398+0000 7f65665a4640 1 -- 172.21.15.97:0/3192186956 >> 172.21.15.97:0/3192186956 conn(0x7f65600b7ea0 msgr2=0x7f65600b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:33.399 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.398+0000 7f65665a4640 1 -- 172.21.15.97:0/3192186956 shutdown_connections 2024-03-25T18:19:33.399 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.398+0000 7f65665a4640 1 -- 172.21.15.97:0/3192186956 wait complete. 2024-03-25T18:19:33.399 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.399+0000 7f65665a4640 10 monclient: build_initial_monmap 2024-03-25T18:19:33.400 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.399+0000 7f65665a4640 10 monclient: monmap: 2024-03-25T18:19:33.400 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:19:33.400 INFO:teuthology.misc.health.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:19:33.400 INFO:teuthology.misc.health.smithi097.stderr:last_changed 0.000000 2024-03-25T18:19:33.400 INFO:teuthology.misc.health.smithi097.stderr:created 0.000000 2024-03-25T18:19:33.400 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:33.400 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 1 2024-03-25T18:19:33.400 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:19:33.400 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T18:19:33.400 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-c 2024-03-25T18:19:33.400 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:19:33.401 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.399+0000 7f65665a4640 1 librados: starting msgr at 2024-03-25T18:19:33.401 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.399+0000 7f65665a4640 1 librados: starting objecter 2024-03-25T18:19:33.401 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.399+0000 7f65665a4640 1 Processor -- start 2024-03-25T18:19:33.401 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.400+0000 7f65665a4640 1 -- start start 2024-03-25T18:19:33.401 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.400+0000 7f65665a4640 1 librados: setting wanted keys 2024-03-25T18:19:33.402 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.400+0000 7f65665a4640 1 librados: calling monclient init 2024-03-25T18:19:33.402 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.400+0000 7f65665a4640 10 monclient: init 2024-03-25T18:19:33.402 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.400+0000 7f65665a4640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:33.402 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.400+0000 7f65665a4640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:19:33.402 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.400+0000 7f65665a4640 10 monclient(hunting): picked mon.noname-b con 0x7f65600a98f0 addr v1:172.21.15.98:6789/0 2024-03-25T18:19:33.402 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.400+0000 7f65665a4640 10 monclient(hunting): picked mon.noname-a con 0x7f656013f710 addr v1:172.21.15.97:6789/0 2024-03-25T18:19:33.402 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.400+0000 7f65665a4640 10 monclient(hunting): picked mon.noname-c con 0x7f65601430c0 addr v1:172.21.15.97:6790/0 2024-03-25T18:19:33.402 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.400+0000 7f65665a4640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65600abac0 con 0x7f656013f710 2024-03-25T18:19:33.402 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.400+0000 7f65665a4640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65600abda0 con 0x7f65600a98f0 2024-03-25T18:19:33.402 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.400+0000 7f65665a4640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65601380e0 con 0x7f65601430c0 2024-03-25T18:19:33.402 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.400+0000 7f65665a4640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:33.402 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.400+0000 7f65665a4640 10 monclient(hunting): authenticate will time out at 7785.675840s 2024-03-25T18:19:33.413 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.413+0000 7f655f7fe640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f656013f710 0x7f6560138b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:47860/0 (socket says 172.21.15.97:47860) 2024-03-25T18:19:33.413 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.413+0000 7f655f7fe640 1 -- 172.21.15.97:0/2326806731 learned_addr learned my addr 172.21.15.97:0/2326806731 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:19:33.414 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.413+0000 7f655cff9640 10 client.?.objecter ms_handle_connect 0x7f656013f710 2024-03-25T18:19:33.414 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.414+0000 7f655cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:33.422 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.422+0000 7f655cff9640 10 client.?.objecter ms_handle_connect 0x7f65601430c0 2024-03-25T18:19:33.422 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.422+0000 7f655cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:33.423 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.423+0000 7f655cff9640 10 client.?.objecter ms_handle_connect 0x7f65600a98f0 2024-03-25T18:19:33.423 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.423+0000 7f655cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:33.432 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.432+0000 7f655cff9640 1 -- 172.21.15.97:0/2326806731 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1455387980 0 0) 0x7f65600abac0 con 0x7f656013f710 2024-03-25T18:19:33.432 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.432+0000 7f655cff9640 10 monclient(hunting): _negotiate 2024-03-25T18:19:33.432 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.432+0000 7f655cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:33.432 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.432+0000 7f655cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:33.432 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.432+0000 7f655cff9640 10 monclient(hunting): my global_id is 4368 2024-03-25T18:19:33.432 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.432+0000 7f655cff9640 1 -- 172.21.15.97:0/2326806731 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f654c004eb0 con 0x7f656013f710 2024-03-25T18:19:33.434 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.433+0000 7f655cff9640 1 -- 172.21.15.97:0/2326806731 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1688486215 0 0) 0x7f65600abda0 con 0x7f65600a98f0 2024-03-25T18:19:33.434 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.433+0000 7f655cff9640 10 monclient(hunting): _negotiate 2024-03-25T18:19:33.434 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.433+0000 7f655cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:33.434 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.433+0000 7f655cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:33.434 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.433+0000 7f655cff9640 10 monclient(hunting): my global_id is 4366 2024-03-25T18:19:33.434 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.433+0000 7f655cff9640 1 -- 172.21.15.97:0/2326806731 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65600abac0 con 0x7f65600a98f0 2024-03-25T18:19:33.439 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.439+0000 7f655cff9640 1 -- 172.21.15.97:0/2326806731 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2558441628 0 0) 0x7f65601380e0 con 0x7f65601430c0 2024-03-25T18:19:33.439 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.439+0000 7f655cff9640 10 monclient(hunting): _negotiate 2024-03-25T18:19:33.439 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.439+0000 7f655cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:33.439 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.439+0000 7f655cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:33.439 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.439+0000 7f655cff9640 10 monclient(hunting): my global_id is 4361 2024-03-25T18:19:33.440 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.439+0000 7f655cff9640 1 -- 172.21.15.97:0/2326806731 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65600abda0 con 0x7f65601430c0 2024-03-25T18:19:33.441 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.441+0000 7f655cff9640 1 -- 172.21.15.97:0/2326806731 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1714196107 0 0) 0x7f654c004eb0 con 0x7f656013f710 2024-03-25T18:19:33.441 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.441+0000 7f655cff9640 1 -- 172.21.15.97:0/2326806731 --> v1:172.21.15.97:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f65601380e0 con 0x7f656013f710 2024-03-25T18:19:33.444 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.444+0000 7f655cff9640 1 -- 172.21.15.97:0/2326806731 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3284380470 0 0) 0x7f65600abac0 con 0x7f65600a98f0 2024-03-25T18:19:33.444 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.444+0000 7f655cff9640 1 -- 172.21.15.97:0/2326806731 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f654c004eb0 con 0x7f65600a98f0 2024-03-25T18:19:33.444 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.444+0000 7f655cff9640 1 -- 172.21.15.97:0/2326806731 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f65500018b0 con 0x7f65600a98f0 2024-03-25T18:19:33.444 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.444+0000 7f655cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:33.444 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.444+0000 7f655cff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:19:33.444 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.444+0000 7f655cff9640 10 monclient(hunting): dump: 2024-03-25T18:19:33.444 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:19:33.444 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:19:33.444 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:19:33.445 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:19:33.445 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:33.445 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:19:33.445 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:19:33.445 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:19:33.445 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:19:33.445 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:19:33.445 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.444+0000 7f655cff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:19:33.449 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.449+0000 7f655cff9640 1 -- 172.21.15.97:0/2326806731 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f6554001cc0 con 0x7f656013f710 2024-03-25T18:19:33.449 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.449+0000 7f655cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:33.450 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.449+0000 7f655cff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:19:33.450 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.449+0000 7f655cff9640 10 monclient(hunting): dump: 2024-03-25T18:19:33.450 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:19:33.450 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:19:33.450 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:19:33.450 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:19:33.450 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:33.450 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:19:33.450 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:19:33.450 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:19:33.450 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:19:33.450 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:19:33.452 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.451+0000 7f655cff9640 1 -- 172.21.15.97:0/2326806731 <== mon.2 v1:172.21.15.97:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1588826184 0 0) 0x7f65600abda0 con 0x7f65601430c0 2024-03-25T18:19:33.452 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.451+0000 7f655cff9640 1 -- 172.21.15.97:0/2326806731 --> v1:172.21.15.97:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f65600abac0 con 0x7f65601430c0 2024-03-25T18:19:33.452 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.451+0000 7f655cff9640 1 -- 172.21.15.97:0/2326806731 <== mon.2 v1:172.21.15.97:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f6558001a60 con 0x7f65601430c0 2024-03-25T18:19:33.452 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.451+0000 7f655cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:33.452 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.451+0000 7f655cff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:19:33.452 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.451+0000 7f655cff9640 10 monclient(hunting): dump: 2024-03-25T18:19:33.452 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:19:33.452 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:19:33.452 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:19:33.452 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:19:33.452 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:33.453 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:19:33.453 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:19:33.453 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:19:33.453 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:19:33.453 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:19:33.455 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.454+0000 7f655cff9640 1 -- 172.21.15.97:0/2326806731 <== mon.0 v1:172.21.15.97:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2382233694 0 0) 0x7f65601380e0 con 0x7f656013f710 2024-03-25T18:19:33.455 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.454+0000 7f655cff9640 1 -- 172.21.15.97:0/2326806731 >> v1:172.21.15.97:6790/0 conn(0x7f65601430c0 legacy=0x7f65601e26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:33.455 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.454+0000 7f655cff9640 1 -- 172.21.15.97:0/2326806731 >> v1:172.21.15.98:6789/0 conn(0x7f65600a98f0 legacy=0x7f65600ac580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:33.455 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.454+0000 7f655cff9640 10 monclient: _finish_hunting 0 2024-03-25T18:19:33.455 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.454+0000 7f655cff9640 1 monclient: found mon.a 2024-03-25T18:19:33.455 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.454+0000 7f655cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:33.455 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.454+0000 7f655cff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:19:33.455 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.454+0000 7f655cff9640 1 -- 172.21.15.97:0/2326806731 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6560137d90 con 0x7f656013f710 2024-03-25T18:19:33.455 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.454+0000 7f655cff9640 10 monclient: _finish_auth 0 2024-03-25T18:19:33.455 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.454+0000 7f655cff9640 10 monclient: _check_auth_tickets 2024-03-25T18:19:33.455 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.454+0000 7f655cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:33.455 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.454+0000 7f65665a4640 5 monclient: authenticate success, global_id 4368 2024-03-25T18:19:33.455 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.454+0000 7f65665a4640 10 monclient: _renew_subs 2024-03-25T18:19:33.455 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.454+0000 7f65665a4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:19:33.456 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.454+0000 7f65665a4640 1 -- 172.21.15.97:0/2326806731 --> v1:172.21.15.97:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6560139240 con 0x7f656013f710 2024-03-25T18:19:33.456 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.454+0000 7f65665a4640 10 client.4368.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:19:33.456 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.454+0000 7f65665a4640 10 monclient: _renew_subs 2024-03-25T18:19:33.456 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.454+0000 7f65665a4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:19:33.456 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.454+0000 7f65665a4640 1 -- 172.21.15.97:0/2326806731 --> v1:172.21.15.97:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6560136830 con 0x7f656013f710 2024-03-25T18:19:33.456 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.454+0000 7f65665a4640 1 librados: init done 2024-03-25T18:19:33.456 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.455+0000 7f65665a4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:33.456 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.455+0000 7f65665a4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:33.457 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.455+0000 7f65665a4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:19:33.457 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.455+0000 7f65665a4640 1 -- 172.21.15.97:0/2326806731 --> v1:172.21.15.97:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65600627b0 con 0x7f656013f710 2024-03-25T18:19:33.774 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.773+0000 7f655cff9640 1 -- 172.21.15.97:0/2326806731 <== mon.0 v1:172.21.15.97:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6554003960 con 0x7f656013f710 2024-03-25T18:19:33.774 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.773+0000 7f655cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:33.774 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.773+0000 7f655cff9640 1 -- 172.21.15.97:0/2326806731 <== mon.0 v1:172.21.15.97:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f6554003d30 con 0x7f656013f710 2024-03-25T18:19:33.774 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.773+0000 7f655cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:33.775 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.773+0000 7f655cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:19:33.775 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.773+0000 7f655cff9640 10 monclient: dump: 2024-03-25T18:19:33.775 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:19:33.775 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:19:33.775 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:19:33.775 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:19:33.775 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:33.775 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:19:33.775 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:19:33.775 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:19:33.775 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:19:33.775 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:19:33.775 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.773+0000 7f655e7fc640 4 set_mon_vals no callback set 2024-03-25T18:19:33.775 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.774+0000 7f655cff9640 1 -- 172.21.15.97:0/2326806731 <== mon.0 v1:172.21.15.97:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2377622137 0 0) 0x7f6554003f50 con 0x7f656013f710 2024-03-25T18:19:33.776 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.775+0000 7f655ffff640 10 monclient: get_auth_request con 0x7f654c06e110 auth_method 0 2024-03-25T18:19:33.776 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.775+0000 7f655cff9640 10 client.4368.objecter ms_handle_connect 0x7f654c06e110 2024-03-25T18:19:33.781 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.780+0000 7f655cff9640 1 -- 172.21.15.97:0/2326806731 <== mon.0 v1:172.21.15.97:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 430711918 0 0) 0x7f6554054940 con 0x7f656013f710 2024-03-25T18:19:33.781 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.780+0000 7f655cff9640 10 client.4368.objecter ms_dispatch 0x7f6560000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:19:33.781 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.780+0000 7f655cff9640 3 client.4368.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:19:33.781 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.780+0000 7f655cff9640 3 client.4368.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:19:33.781 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.780+0000 7f655cff9640 20 client.4368.objecter dump_active .. 0 homeless 2024-03-25T18:19:33.781 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.780+0000 7f655cff9640 1 -- 172.21.15.97:0/2326806731 <== mon.0 v1:172.21.15.97:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6554050450 con 0x7f656013f710 2024-03-25T18:19:33.781 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.780+0000 7f655cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:33.781 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:33.780+0000 7f655cff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:19:34.013 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:34.013+0000 7f65665a4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:34.013 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:34.013+0000 7f65665a4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:34.013 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:34.013+0000 7f65665a4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:19:34.013 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:34.013+0000 7f65665a4640 1 -- 172.21.15.97:0/2326806731 --> v1:172.21.15.97:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f65601393d0 con 0x7f656013f710 2024-03-25T18:19:34.041 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:34.041+0000 7f655cff9640 1 -- 172.21.15.97:0/2326806731 <== mon.0 v1:172.21.15.97:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2169427412) 0x7f6554052380 con 0x7f656013f710 2024-03-25T18:19:34.041 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:34.041+0000 7f655cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:34.042 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:34.041+0000 7f655cff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:19:34.042 INFO:teuthology.misc.health.smithi097.stdout: 2024-03-25T18:19:34.042 INFO:teuthology.misc.health.smithi097.stdout:{"epoch":10,"fsid":"4c3d472c-79b8-486b-9e3d-2738d4306754","created":"2024-03-25T18:16:14.737309+0000","modified":"2024-03-25T18:16:20.357492+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:20.357492+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cb5152b9-262f-45c3-94b8-18d0e81df558","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6daf8793-0838-473d-a7d6-9a0d2894c75d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"136f1493-3b9c-4f43-a2e7-641482d69dae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"397d538c-4f64-4daf-9e8b-662d09f91327","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4f1df08-663f-4bda-93f8-3878ab7c24b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c4b18d70-a3ce-4b15-9b70-eae5525bd761","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8dd4033a-de0e-4f8a-96dc-59f3c1f825b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0077722e-ffa6-4a3d-9891-929d9869b286","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:19:34.045 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:34.043+0000 7f65665a4640 10 librados: watch_flush enter 2024-03-25T18:19:34.045 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:34.043+0000 7f65665a4640 10 librados: watch_flush exit 2024-03-25T18:19:34.045 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:34.043+0000 7f65665a4640 20 client.4368.objecter shutdown clearing up homeless session... 2024-03-25T18:19:34.045 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:34.043+0000 7f65665a4640 10 client.4368.objecter successfully canceled tick 2024-03-25T18:19:34.045 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:34.044+0000 7f65665a4640 1 -- 172.21.15.97:0/2326806731 >> v1:172.21.15.98:6800/800387573 conn(0x7f654c06e110 legacy=0x7f654c0705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:34.045 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:34.044+0000 7f65665a4640 10 monclient: shutdown 2024-03-25T18:19:34.045 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:34.044+0000 7f65665a4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:34.045 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:34.044+0000 7f65665a4640 1 -- 172.21.15.97:0/2326806731 >> v1:172.21.15.97:6789/0 conn(0x7f656013f710 legacy=0x7f6560138b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:34.045 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:34.044+0000 7f65665a4640 1 -- 172.21.15.97:0/2326806731 shutdown_connections 2024-03-25T18:19:34.045 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:34.044+0000 7f65665a4640 1 -- 172.21.15.97:0/2326806731 >> 172.21.15.97:0/2326806731 conn(0x7f65600b7ea0 msgr2=0x7f6560074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:34.045 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:34.044+0000 7f65665a4640 1 -- 172.21.15.97:0/2326806731 shutdown_connections 2024-03-25T18:19:34.046 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:34.044+0000 7f65665a4640 1 -- 172.21.15.97:0/2326806731 wait complete. 2024-03-25T18:19:34.046 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:34.044+0000 7f65665a4640 1 librados: shutdown 2024-03-25T18:19:34.056 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:19:40.862 DEBUG:teuthology.orchestra.run.smithi097:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:19:40.976 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:40.975+0000 7fca55d39640 10 monclient: get_monmap_and_config 2024-03-25T18:19:40.976 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:40.975+0000 7fca55d39640 10 monclient: build_initial_monmap 2024-03-25T18:19:40.976 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:40.975+0000 7fca55d39640 10 monclient: monmap: 2024-03-25T18:19:40.976 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:19:40.976 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:19:40.976 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:19:40.975902+0000 2024-03-25T18:19:40.976 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:19:40.975902+0000 2024-03-25T18:19:40.976 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:40.976 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:19:40.976 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:19:40.976 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T18:19:40.977 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-b 2024-03-25T18:19:40.977 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:19:40.978 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:40.976+0000 7fca55d39640 1 Processor -- start 2024-03-25T18:19:40.978 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:40.976+0000 7fca55d39640 1 -- start start 2024-03-25T18:19:40.978 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:40.976+0000 7fca55d39640 10 monclient: init 2024-03-25T18:19:40.978 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:40.976+0000 7fca55d39640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:40.978 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:40.977+0000 7fca55d39640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:19:40.978 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:40.977+0000 7fca55d39640 10 monclient(hunting): picked mon.noname-a con 0x7fca500b4d60 addr v1:172.21.15.97:6789/0 2024-03-25T18:19:40.979 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:40.977+0000 7fca55d39640 10 monclient(hunting): picked mon.noname-b con 0x7fca500b1730 addr v1:172.21.15.97:6790/0 2024-03-25T18:19:40.979 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:40.977+0000 7fca55d39640 10 monclient(hunting): picked mon.noname-c con 0x7fca5014be90 addr v1:172.21.15.98:6789/0 2024-03-25T18:19:40.979 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:40.977+0000 7fca55d39640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca500af940 con 0x7fca500b4d60 2024-03-25T18:19:40.979 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:40.977+0000 7fca55d39640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca500b0a70 con 0x7fca5014be90 2024-03-25T18:19:40.979 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:40.977+0000 7fca55d39640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca500b0d50 con 0x7fca500b1730 2024-03-25T18:19:40.979 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:40.977+0000 7fca55d39640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:40.979 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:40.977+0000 7fca55d39640 10 monclient(hunting): authenticate will time out at 7793.252066s 2024-03-25T18:19:40.981 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:40.980+0000 7fca4effd640 1 --1- >> v1:172.21.15.97:6790/0 conn(0x7fca500b1730 0x7fca500b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6790/0 says I am v1:172.21.15.97:37572/0 (socket says 172.21.15.97:37572) 2024-03-25T18:19:40.981 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:40.980+0000 7fca4effd640 1 -- 172.21.15.97:0/1201603920 learned_addr learned my addr 172.21.15.97:0/1201603920 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:19:40.993 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:40.992+0000 7fca4e7fc640 1 -- 172.21.15.97:0/1201603920 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3927595464 0 0) 0x7fca500af940 con 0x7fca500b4d60 2024-03-25T18:19:40.993 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:40.992+0000 7fca4e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:19:40.993 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:40.992+0000 7fca4e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:40.993 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:40.992+0000 7fca4e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:40.993 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:40.992+0000 7fca4e7fc640 10 monclient(hunting): my global_id is 4371 2024-03-25T18:19:40.993 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:40.992+0000 7fca4e7fc640 1 -- 172.21.15.97:0/1201603920 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca38003520 con 0x7fca500b4d60 2024-03-25T18:19:40.994 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:40.993+0000 7fca4e7fc640 1 -- 172.21.15.97:0/1201603920 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2304442194 0 0) 0x7fca500b0d50 con 0x7fca500b1730 2024-03-25T18:19:40.994 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:40.993+0000 7fca4e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:19:40.994 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:40.993+0000 7fca4e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:40.994 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:40.993+0000 7fca4e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:40.994 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:40.993+0000 7fca4e7fc640 10 monclient(hunting): my global_id is 4364 2024-03-25T18:19:40.994 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:40.993+0000 7fca4e7fc640 1 -- 172.21.15.97:0/1201603920 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca500af940 con 0x7fca500b1730 2024-03-25T18:19:40.999 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:40.999+0000 7fca4e7fc640 1 -- 172.21.15.97:0/1201603920 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 115030441 0 0) 0x7fca38003520 con 0x7fca500b4d60 2024-03-25T18:19:41.000 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:40.999+0000 7fca4e7fc640 1 -- 172.21.15.97:0/1201603920 >> v1:172.21.15.97:6790/0 conn(0x7fca500b1730 legacy=0x7fca500b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:41.000 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:40.999+0000 7fca4e7fc640 1 -- 172.21.15.97:0/1201603920 >> v1:172.21.15.98:6789/0 conn(0x7fca5014be90 legacy=0x7fca5014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:41.000 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:40.999+0000 7fca4e7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:19:41.000 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:40.999+0000 7fca4e7fc640 1 monclient: found mon.noname-a 2024-03-25T18:19:41.000 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:40.999+0000 7fca4e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:41.000 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:40.999+0000 7fca4e7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.97:6789/0 2024-03-25T18:19:41.000 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:40.999+0000 7fca4e7fc640 1 -- 172.21.15.97:0/1201603920 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca500b1030 con 0x7fca500b4d60 2024-03-25T18:19:41.000 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:40.999+0000 7fca4e7fc640 10 monclient: _finish_auth 0 2024-03-25T18:19:41.000 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:40.999+0000 7fca4e7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:19:41.000 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:40.999+0000 7fca4e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:41.000 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:40.999+0000 7fca4e7fc640 1 -- 172.21.15.97:0/1201603920 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7fca40001690 con 0x7fca500b4d60 2024-03-25T18:19:41.001 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:40.999+0000 7fca4e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:41.001 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:40.999+0000 7fca4e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:19:41.001 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:40.999+0000 7fca4e7fc640 10 monclient: dump: 2024-03-25T18:19:41.001 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:19:41.001 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:19:41.001 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:19:41.001 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:19:41.001 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:41.001 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:19:41.001 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:19:41.001 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:19:41.002 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:19:41.002 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:19:41.002 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:40.999+0000 7fca55d39640 5 monclient: authenticate success, global_id 4371 2024-03-25T18:19:41.002 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:40.999+0000 7fca55d39640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:41.012 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:41.012+0000 7fca4e7fc640 1 -- 172.21.15.97:0/1201603920 <== mon.0 v1:172.21.15.97:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fca400017f0 con 0x7fca500b4d60 2024-03-25T18:19:41.012 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:41.012+0000 7fca4e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:41.012 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:41.012+0000 7fca4e7fc640 1 -- 172.21.15.97:0/1201603920 <== mon.0 v1:172.21.15.97:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7fca40001c00 con 0x7fca500b4d60 2024-03-25T18:19:41.012 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:41.012+0000 7fca55d39640 10 monclient: get_monmap_and_config success 2024-03-25T18:19:41.012 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:41.012+0000 7fca55d39640 4 set_mon_vals no callback set 2024-03-25T18:19:41.012 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:41.012+0000 7fca4e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:41.012 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:41.012+0000 7fca4e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:19:41.013 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:41.012+0000 7fca4e7fc640 10 monclient: dump: 2024-03-25T18:19:41.013 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:19:41.013 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:19:41.013 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:19:41.013 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:19:41.013 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:41.013 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:19:41.013 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:19:41.013 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:19:41.013 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:19:41.013 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:19:41.015 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:41.012+0000 7fca55d39640 10 monclient: shutdown 2024-03-25T18:19:41.015 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:41.012+0000 7fca55d39640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:41.015 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:41.012+0000 7fca55d39640 1 -- 172.21.15.97:0/1201603920 >> v1:172.21.15.97:6789/0 conn(0x7fca500b4d60 legacy=0x7fca5013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:41.015 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:41.013+0000 7fca55d39640 1 -- 172.21.15.97:0/1201603920 shutdown_connections 2024-03-25T18:19:41.015 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:41.013+0000 7fca55d39640 1 -- 172.21.15.97:0/1201603920 >> 172.21.15.97:0/1201603920 conn(0x7fca500617a0 msgr2=0x7fca50061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:41.015 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:41.013+0000 7fca55d39640 1 -- 172.21.15.97:0/1201603920 shutdown_connections 2024-03-25T18:19:41.015 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:41.013+0000 7fca55d39640 1 -- 172.21.15.97:0/1201603920 wait complete. 2024-03-25T18:19:41.015 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:41.014+0000 7fca55d39640 10 monclient: build_initial_monmap 2024-03-25T18:19:41.016 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:41.014+0000 7fca55d39640 10 monclient: monmap: 2024-03-25T18:19:41.016 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:19:41.016 INFO:teuthology.misc.health.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:19:41.016 INFO:teuthology.misc.health.smithi097.stderr:last_changed 0.000000 2024-03-25T18:19:41.016 INFO:teuthology.misc.health.smithi097.stderr:created 0.000000 2024-03-25T18:19:41.016 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:41.016 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 1 2024-03-25T18:19:41.016 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:19:41.016 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T18:19:41.016 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-c 2024-03-25T18:19:41.016 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:19:41.017 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:41.014+0000 7fca55d39640 1 librados: starting msgr at 2024-03-25T18:19:41.017 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:41.014+0000 7fca55d39640 1 librados: starting objecter 2024-03-25T18:19:41.017 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:41.014+0000 7fca55d39640 1 Processor -- start 2024-03-25T18:19:41.017 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:41.014+0000 7fca55d39640 1 -- start start 2024-03-25T18:19:41.017 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:41.014+0000 7fca55d39640 1 librados: setting wanted keys 2024-03-25T18:19:41.017 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:41.014+0000 7fca55d39640 1 librados: calling monclient init 2024-03-25T18:19:41.017 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:41.014+0000 7fca55d39640 10 monclient: init 2024-03-25T18:19:41.019 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:41.015+0000 7fca55d39640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:41.019 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:41.015+0000 7fca55d39640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:19:41.019 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:41.015+0000 7fca55d39640 10 monclient(hunting): picked mon.noname-b con 0x7fca500b1730 addr v1:172.21.15.98:6789/0 2024-03-25T18:19:41.019 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:41.016+0000 7fca55d39640 10 monclient(hunting): picked mon.noname-a con 0x7fca500b4d60 addr v1:172.21.15.97:6789/0 2024-03-25T18:19:41.019 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:41.016+0000 7fca55d39640 10 monclient(hunting): picked mon.noname-c con 0x7fca5014be90 addr v1:172.21.15.97:6790/0 2024-03-25T18:19:41.019 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:41.016+0000 7fca55d39640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca50142030 con 0x7fca500b4d60 2024-03-25T18:19:41.019 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:41.016+0000 7fca55d39640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca5013bdf0 con 0x7fca500b1730 2024-03-25T18:19:41.019 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:41.016+0000 7fca55d39640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca501424c0 con 0x7fca5014be90 2024-03-25T18:19:41.019 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:41.016+0000 7fca55d39640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:41.019 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:41.016+0000 7fca55d39640 10 monclient(hunting): authenticate will time out at 7793.291356s 2024-03-25T18:19:41.027 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:41.026+0000 7fca4effd640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fca500b4d60 0x7fca501db110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:42208/0 (socket says 172.21.15.97:42208) 2024-03-25T18:19:41.027 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:41.026+0000 7fca4effd640 1 -- 172.21.15.97:0/554473850 learned_addr learned my addr 172.21.15.97:0/554473850 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:19:41.027 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:41.027+0000 7fca37fff640 10 client.?.objecter ms_handle_connect 0x7fca500b4d60 2024-03-25T18:19:41.027 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:41.027+0000 7fca37fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:41.036 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:41.035+0000 7fca37fff640 1 -- 172.21.15.97:0/554473850 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3445048990 0 0) 0x7fca50142030 con 0x7fca500b4d60 2024-03-25T18:19:41.036 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:41.035+0000 7fca37fff640 10 monclient(hunting): _negotiate 2024-03-25T18:19:41.036 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:41.035+0000 7fca37fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:41.036 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:41.035+0000 7fca37fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:41.036 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:41.035+0000 7fca37fff640 10 monclient(hunting): my global_id is 4374 2024-03-25T18:19:41.036 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:41.035+0000 7fca37fff640 1 -- 172.21.15.97:0/554473850 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca3c004eb0 con 0x7fca500b4d60 2024-03-25T18:19:41.037 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:41.037+0000 7fca37fff640 10 client.?.objecter ms_handle_connect 0x7fca500b1730 2024-03-25T18:19:41.038 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:41.037+0000 7fca37fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:41.045 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:41.044+0000 7fca37fff640 1 -- 172.21.15.97:0/554473850 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1226389480 0 0) 0x7fca5013bdf0 con 0x7fca500b1730 2024-03-25T18:19:41.045 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:41.044+0000 7fca37fff640 10 monclient(hunting): _negotiate 2024-03-25T18:19:41.045 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:41.044+0000 7fca37fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:41.045 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:41.044+0000 7fca37fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:41.045 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:41.044+0000 7fca37fff640 10 monclient(hunting): my global_id is 4372 2024-03-25T18:19:41.045 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:41.044+0000 7fca37fff640 1 -- 172.21.15.97:0/554473850 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca50142030 con 0x7fca500b1730 2024-03-25T18:19:41.045 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:41.045+0000 7fca37fff640 10 client.?.objecter ms_handle_connect 0x7fca5014be90 2024-03-25T18:19:41.046 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:41.045+0000 7fca37fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:41.052 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:41.052+0000 7fca37fff640 1 -- 172.21.15.97:0/554473850 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1765108430 0 0) 0x7fca501424c0 con 0x7fca5014be90 2024-03-25T18:19:41.052 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:41.052+0000 7fca37fff640 10 monclient(hunting): _negotiate 2024-03-25T18:19:41.052 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:41.052+0000 7fca37fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:41.052 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:41.052+0000 7fca37fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:41.053 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:41.052+0000 7fca37fff640 10 monclient(hunting): my global_id is 4367 2024-03-25T18:19:41.053 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:41.052+0000 7fca37fff640 1 -- 172.21.15.97:0/554473850 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca5013bdf0 con 0x7fca5014be90 2024-03-25T18:19:41.060 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:41.059+0000 7fca37fff640 1 -- 172.21.15.97:0/554473850 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1264214789 0 0) 0x7fca50142030 con 0x7fca500b1730 2024-03-25T18:19:41.060 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:41.060+0000 7fca37fff640 1 -- 172.21.15.97:0/554473850 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fca501424c0 con 0x7fca500b1730 2024-03-25T18:19:41.060 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:41.060+0000 7fca37fff640 1 -- 172.21.15.97:0/554473850 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7fca40001b40 con 0x7fca500b1730 2024-03-25T18:19:41.060 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:41.060+0000 7fca37fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:41.060 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:41.060+0000 7fca37fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:19:41.060 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:41.060+0000 7fca37fff640 10 monclient(hunting): dump: 2024-03-25T18:19:41.061 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:19:41.061 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:19:41.061 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:19:41.061 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:19:41.061 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:41.061 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:19:41.061 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:19:41.061 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:19:41.061 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:19:41.061 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:19:41.061 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:41.060+0000 7fca37fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:19:41.068 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:41.067+0000 7fca37fff640 1 -- 172.21.15.97:0/554473850 <== mon.2 v1:172.21.15.97:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2191871920 0 0) 0x7fca5013bdf0 con 0x7fca5014be90 2024-03-25T18:19:41.068 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:41.067+0000 7fca37fff640 1 -- 172.21.15.97:0/554473850 --> v1:172.21.15.97:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fca50142030 con 0x7fca5014be90 2024-03-25T18:19:41.068 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:41.067+0000 7fca37fff640 1 -- 172.21.15.97:0/554473850 <== mon.2 v1:172.21.15.97:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7fca480019e0 con 0x7fca5014be90 2024-03-25T18:19:41.068 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:41.067+0000 7fca37fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:41.068 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:41.067+0000 7fca37fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:19:41.068 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:41.067+0000 7fca37fff640 10 monclient(hunting): dump: 2024-03-25T18:19:41.068 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:19:41.068 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:19:41.068 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:19:41.068 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:19:41.069 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:41.069 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:19:41.069 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:19:41.069 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:19:41.069 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:19:41.069 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:19:41.075 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:41.075+0000 7fca37fff640 1 -- 172.21.15.97:0/554473850 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1648051524 0 0) 0x7fca3c004eb0 con 0x7fca500b4d60 2024-03-25T18:19:41.075 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:41.075+0000 7fca37fff640 1 -- 172.21.15.97:0/554473850 --> v1:172.21.15.97:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fca5013bdf0 con 0x7fca500b4d60 2024-03-25T18:19:41.075 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:41.075+0000 7fca37fff640 1 -- 172.21.15.97:0/554473850 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7fca440019b0 con 0x7fca500b4d60 2024-03-25T18:19:41.075 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:41.075+0000 7fca37fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:41.076 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:41.075+0000 7fca37fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:19:41.076 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:41.075+0000 7fca37fff640 10 monclient(hunting): dump: 2024-03-25T18:19:41.076 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:19:41.076 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:19:41.076 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:19:41.076 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:19:41.076 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:41.076 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:19:41.076 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:19:41.076 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:19:41.076 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:19:41.076 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:19:41.080 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:41.080+0000 7fca37fff640 1 -- 172.21.15.97:0/554473850 <== mon.2 v1:172.21.15.97:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1825520289 0 0) 0x7fca50142030 con 0x7fca5014be90 2024-03-25T18:19:41.081 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:41.080+0000 7fca37fff640 1 -- 172.21.15.97:0/554473850 >> v1:172.21.15.98:6789/0 conn(0x7fca500b1730 legacy=0x7fca50140840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:41.081 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:41.080+0000 7fca37fff640 1 -- 172.21.15.97:0/554473850 >> v1:172.21.15.97:6789/0 conn(0x7fca500b4d60 legacy=0x7fca501db110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:41.081 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:41.080+0000 7fca37fff640 10 monclient: _finish_hunting 0 2024-03-25T18:19:41.081 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:41.080+0000 7fca37fff640 1 monclient: found mon.c 2024-03-25T18:19:41.081 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:41.080+0000 7fca37fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:41.081 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:41.080+0000 7fca37fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.97:6790/0 2024-03-25T18:19:41.081 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:41.080+0000 7fca37fff640 1 -- 172.21.15.97:0/554473850 --> v1:172.21.15.97:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca50142650 con 0x7fca5014be90 2024-03-25T18:19:41.081 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:41.080+0000 7fca37fff640 10 monclient: _finish_auth 0 2024-03-25T18:19:41.081 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:41.080+0000 7fca37fff640 10 monclient: _check_auth_tickets 2024-03-25T18:19:41.081 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:41.080+0000 7fca37fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:41.081 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:41.080+0000 7fca55d39640 5 monclient: authenticate success, global_id 4367 2024-03-25T18:19:41.081 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:41.080+0000 7fca55d39640 10 monclient: _renew_subs 2024-03-25T18:19:41.082 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:41.080+0000 7fca55d39640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.97:6790/0 2024-03-25T18:19:41.082 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:41.080+0000 7fca55d39640 1 -- 172.21.15.97:0/554473850 --> v1:172.21.15.97:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca501db820 con 0x7fca5014be90 2024-03-25T18:19:41.082 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:41.080+0000 7fca55d39640 10 client.4367.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:19:41.082 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:41.080+0000 7fca55d39640 10 monclient: _renew_subs 2024-03-25T18:19:41.082 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:41.080+0000 7fca55d39640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.97:6790/0 2024-03-25T18:19:41.082 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:41.080+0000 7fca55d39640 1 -- 172.21.15.97:0/554473850 --> v1:172.21.15.97:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fca501435a0 con 0x7fca5014be90 2024-03-25T18:19:41.082 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:41.080+0000 7fca55d39640 1 librados: init done 2024-03-25T18:19:41.083 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:41.081+0000 7fca55d39640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:41.083 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:41.081+0000 7fca55d39640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:41.083 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:41.081+0000 7fca55d39640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.97:6790/0 2024-03-25T18:19:41.083 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:41.081+0000 7fca55d39640 1 -- 172.21.15.97:0/554473850 --> v1:172.21.15.97:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca501daf60 con 0x7fca5014be90 2024-03-25T18:19:41.377 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:41.376+0000 7fca37fff640 1 -- 172.21.15.97:0/554473850 <== mon.2 v1:172.21.15.97:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fca48001f10 con 0x7fca5014be90 2024-03-25T18:19:41.378 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:41.376+0000 7fca37fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:41.378 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:41.376+0000 7fca37fff640 1 -- 172.21.15.97:0/554473850 <== mon.2 v1:172.21.15.97:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7fca48003b10 con 0x7fca5014be90 2024-03-25T18:19:41.378 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:41.376+0000 7fca37fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:41.378 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:41.377+0000 7fca37fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:19:41.378 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:41.377+0000 7fca37fff640 10 monclient: dump: 2024-03-25T18:19:41.378 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:19:41.378 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:19:41.379 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:19:41.379 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:19:41.379 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:41.379 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:19:41.379 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:19:41.379 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:19:41.379 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:19:41.379 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:19:41.379 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:41.376+0000 7fca4dffb640 4 set_mon_vals no callback set 2024-03-25T18:19:41.380 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:41.377+0000 7fca37fff640 1 -- 172.21.15.97:0/554473850 <== mon.2 v1:172.21.15.97:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2377622137 0 0) 0x7fca48003d30 con 0x7fca5014be90 2024-03-25T18:19:41.380 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:41.378+0000 7fca4f7fe640 10 monclient: get_auth_request con 0x7fca3c06e160 auth_method 0 2024-03-25T18:19:41.380 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:41.379+0000 7fca37fff640 10 client.4367.objecter ms_handle_connect 0x7fca3c06e160 2024-03-25T18:19:41.386 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:41.385+0000 7fca37fff640 1 -- 172.21.15.97:0/554473850 <== mon.2 v1:172.21.15.97:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 430711918 0 0) 0x7fca48053630 con 0x7fca5014be90 2024-03-25T18:19:41.386 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:41.385+0000 7fca37fff640 10 client.4367.objecter ms_dispatch 0x7fca50000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:19:41.386 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:41.385+0000 7fca37fff640 3 client.4367.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:19:41.386 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:41.385+0000 7fca37fff640 3 client.4367.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:19:41.386 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:41.385+0000 7fca37fff640 20 client.4367.objecter dump_active .. 0 homeless 2024-03-25T18:19:41.386 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:41.385+0000 7fca37fff640 1 -- 172.21.15.97:0/554473850 <== mon.2 v1:172.21.15.97:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fca4804f140 con 0x7fca5014be90 2024-03-25T18:19:41.386 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:41.385+0000 7fca37fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:41.387 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:41.385+0000 7fca37fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:19:41.614 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:41.613+0000 7fca55d39640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:41.614 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:41.613+0000 7fca55d39640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:41.614 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:41.613+0000 7fca55d39640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.97:6790/0 2024-03-25T18:19:41.614 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:41.613+0000 7fca55d39640 1 -- 172.21.15.97:0/554473850 --> v1:172.21.15.97:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fca501401f0 con 0x7fca5014be90 2024-03-25T18:19:41.680 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:41.680+0000 7fca37fff640 1 -- 172.21.15.97:0/554473850 <== mon.2 v1:172.21.15.97:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2169427412) 0x7fca48051070 con 0x7fca5014be90 2024-03-25T18:19:41.680 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:41.680+0000 7fca37fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:41.680 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:41.680+0000 7fca37fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:19:41.681 INFO:teuthology.misc.health.smithi097.stdout: 2024-03-25T18:19:41.681 INFO:teuthology.misc.health.smithi097.stdout:{"epoch":10,"fsid":"4c3d472c-79b8-486b-9e3d-2738d4306754","created":"2024-03-25T18:16:14.737309+0000","modified":"2024-03-25T18:16:20.357492+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:20.357492+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cb5152b9-262f-45c3-94b8-18d0e81df558","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6daf8793-0838-473d-a7d6-9a0d2894c75d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"136f1493-3b9c-4f43-a2e7-641482d69dae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"397d538c-4f64-4daf-9e8b-662d09f91327","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4f1df08-663f-4bda-93f8-3878ab7c24b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c4b18d70-a3ce-4b15-9b70-eae5525bd761","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8dd4033a-de0e-4f8a-96dc-59f3c1f825b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0077722e-ffa6-4a3d-9891-929d9869b286","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:19:41.683 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:41.682+0000 7fca55d39640 10 librados: watch_flush enter 2024-03-25T18:19:41.683 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:41.682+0000 7fca55d39640 10 librados: watch_flush exit 2024-03-25T18:19:41.683 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:41.682+0000 7fca55d39640 20 client.4367.objecter shutdown clearing up homeless session... 2024-03-25T18:19:41.683 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:41.682+0000 7fca55d39640 10 client.4367.objecter successfully canceled tick 2024-03-25T18:19:41.683 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:41.682+0000 7fca55d39640 1 -- 172.21.15.97:0/554473850 >> v1:172.21.15.98:6800/800387573 conn(0x7fca3c06e160 legacy=0x7fca3c070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:41.684 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:41.682+0000 7fca55d39640 10 monclient: shutdown 2024-03-25T18:19:41.684 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:41.682+0000 7fca55d39640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:41.684 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:41.682+0000 7fca55d39640 1 -- 172.21.15.97:0/554473850 >> v1:172.21.15.97:6790/0 conn(0x7fca5014be90 legacy=0x7fca50142cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:41.684 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:41.683+0000 7fca55d39640 1 -- 172.21.15.97:0/554473850 shutdown_connections 2024-03-25T18:19:41.684 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:41.683+0000 7fca55d39640 1 -- 172.21.15.97:0/554473850 >> 172.21.15.97:0/554473850 conn(0x7fca500617a0 msgr2=0x7fca5013f430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:41.684 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:41.683+0000 7fca55d39640 1 -- 172.21.15.97:0/554473850 shutdown_connections 2024-03-25T18:19:41.684 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:41.683+0000 7fca55d39640 1 -- 172.21.15.97:0/554473850 wait complete. 2024-03-25T18:19:41.684 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:41.683+0000 7fca55d39640 1 librados: shutdown 2024-03-25T18:19:41.695 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:19:48.503 DEBUG:teuthology.orchestra.run.smithi097:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:19:48.620 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:48.619+0000 7f9b501af640 10 monclient: get_monmap_and_config 2024-03-25T18:19:48.620 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:48.619+0000 7f9b501af640 10 monclient: build_initial_monmap 2024-03-25T18:19:48.620 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:48.619+0000 7f9b501af640 10 monclient: monmap: 2024-03-25T18:19:48.620 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:19:48.620 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:19:48.620 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:19:48.620357+0000 2024-03-25T18:19:48.620 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:19:48.620357+0000 2024-03-25T18:19:48.620 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:48.620 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:19:48.620 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:19:48.621 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T18:19:48.621 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-b 2024-03-25T18:19:48.621 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:19:48.622 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:48.619+0000 7f9b501af640 1 Processor -- start 2024-03-25T18:19:48.622 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:48.619+0000 7f9b501af640 1 -- start start 2024-03-25T18:19:48.622 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:48.619+0000 7f9b501af640 10 monclient: init 2024-03-25T18:19:48.622 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:48.620+0000 7f9b501af640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:48.622 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:48.620+0000 7f9b501af640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:19:48.622 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:48.620+0000 7f9b501af640 10 monclient(hunting): picked mon.noname-b con 0x7f9b480b4d60 addr v1:172.21.15.97:6790/0 2024-03-25T18:19:48.622 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:48.620+0000 7f9b501af640 10 monclient(hunting): picked mon.noname-c con 0x7f9b480b1730 addr v1:172.21.15.98:6789/0 2024-03-25T18:19:48.622 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:48.620+0000 7f9b501af640 10 monclient(hunting): picked mon.noname-a con 0x7f9b4814be90 addr v1:172.21.15.97:6789/0 2024-03-25T18:19:48.623 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:48.620+0000 7f9b501af640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b480af940 con 0x7f9b4814be90 2024-03-25T18:19:48.623 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:48.620+0000 7f9b501af640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b480b0a70 con 0x7f9b480b1730 2024-03-25T18:19:48.623 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:48.620+0000 7f9b501af640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b480b0d50 con 0x7f9b480b4d60 2024-03-25T18:19:48.623 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:48.620+0000 7f9b501af640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:48.623 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:48.620+0000 7f9b501af640 10 monclient(hunting): authenticate will time out at 7800.896802s 2024-03-25T18:19:48.629 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:48.628+0000 7f9b4df4b640 1 --1- >> v1:172.21.15.97:6790/0 conn(0x7f9b480b4d60 0x7f9b4813f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6790/0 says I am v1:172.21.15.97:42432/0 (socket says 172.21.15.97:42432) 2024-03-25T18:19:48.629 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:48.628+0000 7f9b4df4b640 1 -- 172.21.15.97:0/2891739992 learned_addr learned my addr 172.21.15.97:0/2891739992 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:19:48.645 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:48.644+0000 7f9b4cf49640 1 -- 172.21.15.97:0/2891739992 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1724177599 0 0) 0x7f9b480b0a70 con 0x7f9b480b1730 2024-03-25T18:19:48.646 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:48.644+0000 7f9b4cf49640 10 monclient(hunting): _negotiate 2024-03-25T18:19:48.646 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:48.644+0000 7f9b4cf49640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:48.646 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:48.644+0000 7f9b4cf49640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:48.646 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:48.644+0000 7f9b4cf49640 10 monclient(hunting): my global_id is 4375 2024-03-25T18:19:48.646 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:48.644+0000 7f9b4cf49640 1 -- 172.21.15.97:0/2891739992 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b34003520 con 0x7f9b480b1730 2024-03-25T18:19:48.648 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:48.646+0000 7f9b4cf49640 1 -- 172.21.15.97:0/2891739992 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1262020938 0 0) 0x7f9b480b0d50 con 0x7f9b480b4d60 2024-03-25T18:19:48.648 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:48.646+0000 7f9b4cf49640 10 monclient(hunting): _negotiate 2024-03-25T18:19:48.648 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:48.646+0000 7f9b4cf49640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:48.648 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:48.646+0000 7f9b4cf49640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:48.648 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:48.646+0000 7f9b4cf49640 10 monclient(hunting): my global_id is 4370 2024-03-25T18:19:48.648 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:48.646+0000 7f9b4cf49640 1 -- 172.21.15.97:0/2891739992 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b480b0a70 con 0x7f9b480b4d60 2024-03-25T18:19:48.665 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:48.664+0000 7f9b4cf49640 1 -- 172.21.15.97:0/2891739992 <== mon.2 v1:172.21.15.97:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1708601414 0 0) 0x7f9b480b0a70 con 0x7f9b480b4d60 2024-03-25T18:19:48.666 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:48.664+0000 7f9b4cf49640 1 -- 172.21.15.97:0/2891739992 >> v1:172.21.15.98:6789/0 conn(0x7f9b480b1730 legacy=0x7f9b480b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:48.666 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:48.664+0000 7f9b4cf49640 1 -- 172.21.15.97:0/2891739992 >> v1:172.21.15.97:6789/0 conn(0x7f9b4814be90 legacy=0x7f9b4814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:48.666 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:48.664+0000 7f9b4cf49640 10 monclient: _finish_hunting 0 2024-03-25T18:19:48.666 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:48.664+0000 7f9b4cf49640 1 monclient: found mon.noname-b 2024-03-25T18:19:48.666 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:48.664+0000 7f9b4cf49640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:48.666 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:48.664+0000 7f9b4cf49640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.97:6790/0 2024-03-25T18:19:48.666 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:48.664+0000 7f9b4cf49640 1 -- 172.21.15.97:0/2891739992 --> v1:172.21.15.97:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b480b1030 con 0x7f9b480b4d60 2024-03-25T18:19:48.666 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:48.664+0000 7f9b4cf49640 10 monclient: _finish_auth 0 2024-03-25T18:19:48.666 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:48.664+0000 7f9b4cf49640 10 monclient: _check_auth_tickets 2024-03-25T18:19:48.666 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:48.664+0000 7f9b4cf49640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:48.666 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:48.664+0000 7f9b4cf49640 1 -- 172.21.15.97:0/2891739992 <== mon.2 v1:172.21.15.97:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f9b44001670 con 0x7f9b480b4d60 2024-03-25T18:19:48.666 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:48.664+0000 7f9b4cf49640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:48.666 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:48.664+0000 7f9b4cf49640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:19:48.667 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:48.664+0000 7f9b4cf49640 10 monclient: dump: 2024-03-25T18:19:48.667 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:19:48.667 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:19:48.667 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:19:48.667 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:19:48.667 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:48.667 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:19:48.667 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:19:48.667 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:19:48.667 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:19:48.667 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:19:48.667 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:48.664+0000 7f9b501af640 5 monclient: authenticate success, global_id 4370 2024-03-25T18:19:48.667 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:48.664+0000 7f9b501af640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:48.683 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:48.681+0000 7f9b4cf49640 1 -- 172.21.15.97:0/2891739992 <== mon.2 v1:172.21.15.97:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9b440017d0 con 0x7f9b480b4d60 2024-03-25T18:19:48.683 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:48.681+0000 7f9b4cf49640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:48.683 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:48.681+0000 7f9b4cf49640 1 -- 172.21.15.97:0/2891739992 <== mon.2 v1:172.21.15.97:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f9b44001ba0 con 0x7f9b480b4d60 2024-03-25T18:19:48.683 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:48.681+0000 7f9b4cf49640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:48.683 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:48.681+0000 7f9b4cf49640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:19:48.683 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:48.681+0000 7f9b4cf49640 10 monclient: dump: 2024-03-25T18:19:48.683 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:19:48.683 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:19:48.683 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:19:48.683 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:19:48.683 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:48.683 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:19:48.683 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:19:48.683 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:19:48.684 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:19:48.684 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:19:48.684 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:48.681+0000 7f9b501af640 10 monclient: get_monmap_and_config success 2024-03-25T18:19:48.684 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:48.681+0000 7f9b501af640 4 set_mon_vals no callback set 2024-03-25T18:19:48.685 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:48.682+0000 7f9b501af640 10 monclient: shutdown 2024-03-25T18:19:48.685 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:48.682+0000 7f9b501af640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:48.685 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:48.682+0000 7f9b501af640 1 -- 172.21.15.97:0/2891739992 >> v1:172.21.15.97:6790/0 conn(0x7f9b480b4d60 legacy=0x7f9b4813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:48.685 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:48.682+0000 7f9b501af640 1 -- 172.21.15.97:0/2891739992 shutdown_connections 2024-03-25T18:19:48.685 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:48.682+0000 7f9b501af640 1 -- 172.21.15.97:0/2891739992 >> 172.21.15.97:0/2891739992 conn(0x7f9b480617a0 msgr2=0x7f9b48061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:48.685 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:48.682+0000 7f9b501af640 1 -- 172.21.15.97:0/2891739992 shutdown_connections 2024-03-25T18:19:48.685 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:48.682+0000 7f9b501af640 1 -- 172.21.15.97:0/2891739992 wait complete. 2024-03-25T18:19:48.685 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:48.683+0000 7f9b501af640 10 monclient: build_initial_monmap 2024-03-25T18:19:48.685 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:48.683+0000 7f9b501af640 10 monclient: monmap: 2024-03-25T18:19:48.685 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:19:48.686 INFO:teuthology.misc.health.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:19:48.686 INFO:teuthology.misc.health.smithi097.stderr:last_changed 0.000000 2024-03-25T18:19:48.686 INFO:teuthology.misc.health.smithi097.stderr:created 0.000000 2024-03-25T18:19:48.686 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:48.686 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 1 2024-03-25T18:19:48.686 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:19:48.686 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T18:19:48.686 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-c 2024-03-25T18:19:48.686 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:19:48.686 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:48.683+0000 7f9b501af640 1 librados: starting msgr at 2024-03-25T18:19:48.686 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:48.683+0000 7f9b501af640 1 librados: starting objecter 2024-03-25T18:19:48.686 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:48.683+0000 7f9b501af640 1 Processor -- start 2024-03-25T18:19:48.686 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:48.683+0000 7f9b501af640 1 -- start start 2024-03-25T18:19:48.686 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:48.683+0000 7f9b501af640 1 librados: setting wanted keys 2024-03-25T18:19:48.687 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:48.683+0000 7f9b501af640 1 librados: calling monclient init 2024-03-25T18:19:48.687 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:48.683+0000 7f9b501af640 10 monclient: init 2024-03-25T18:19:48.688 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:48.684+0000 7f9b501af640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:48.688 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:48.684+0000 7f9b501af640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:19:48.688 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:48.684+0000 7f9b501af640 10 monclient(hunting): picked mon.noname-c con 0x7f9b480b1730 addr v1:172.21.15.97:6790/0 2024-03-25T18:19:48.688 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:48.684+0000 7f9b501af640 10 monclient(hunting): picked mon.noname-a con 0x7f9b480b4d60 addr v1:172.21.15.97:6789/0 2024-03-25T18:19:48.688 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:48.684+0000 7f9b501af640 10 monclient(hunting): picked mon.noname-b con 0x7f9b4814be90 addr v1:172.21.15.98:6789/0 2024-03-25T18:19:48.688 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:48.684+0000 7f9b501af640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b48137300 con 0x7f9b480b4d60 2024-03-25T18:19:48.688 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:48.684+0000 7f9b501af640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b48136f40 con 0x7f9b4814be90 2024-03-25T18:19:48.688 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:48.684+0000 7f9b501af640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b48143d90 con 0x7f9b480b1730 2024-03-25T18:19:48.688 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:48.684+0000 7f9b501af640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:48.688 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:48.684+0000 7f9b501af640 10 monclient(hunting): authenticate will time out at 7800.960484s 2024-03-25T18:19:48.688 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:48.687+0000 7f9b4e74c640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f9b4814be90 0x7f9b481db620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.97:46908/0 (socket says 172.21.15.97:46908) 2024-03-25T18:19:48.689 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:48.687+0000 7f9b4e74c640 1 -- 172.21.15.97:0/1821521763 learned_addr learned my addr 172.21.15.97:0/1821521763 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:19:48.691 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:48.690+0000 7f9b3e7fc640 10 client.?.objecter ms_handle_connect 0x7f9b4814be90 2024-03-25T18:19:48.691 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:48.690+0000 7f9b3e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:48.701 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:48.700+0000 7f9b3e7fc640 10 client.?.objecter ms_handle_connect 0x7f9b480b1730 2024-03-25T18:19:48.701 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:48.700+0000 7f9b3e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:48.701 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:48.700+0000 7f9b3e7fc640 1 -- 172.21.15.97:0/1821521763 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1096958225 0 0) 0x7f9b48136f40 con 0x7f9b4814be90 2024-03-25T18:19:48.701 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:48.700+0000 7f9b3e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:19:48.701 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:48.700+0000 7f9b3e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:48.702 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:48.700+0000 7f9b3e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:48.702 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:48.700+0000 7f9b3e7fc640 10 monclient(hunting): my global_id is 4378 2024-03-25T18:19:48.702 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:48.700+0000 7f9b3e7fc640 1 -- 172.21.15.97:0/1821521763 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b2c004eb0 con 0x7f9b4814be90 2024-03-25T18:19:48.702 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:48.701+0000 7f9b3e7fc640 10 client.?.objecter ms_handle_connect 0x7f9b480b4d60 2024-03-25T18:19:48.702 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:48.701+0000 7f9b3e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:48.710 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:48.709+0000 7f9b3e7fc640 1 -- 172.21.15.97:0/1821521763 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 748219597 0 0) 0x7f9b48143d90 con 0x7f9b480b1730 2024-03-25T18:19:48.711 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:48.709+0000 7f9b3e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:19:48.711 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:48.709+0000 7f9b3e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:48.711 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:48.709+0000 7f9b3e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:48.711 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:48.709+0000 7f9b3e7fc640 10 monclient(hunting): my global_id is 4373 2024-03-25T18:19:48.711 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:48.709+0000 7f9b3e7fc640 1 -- 172.21.15.97:0/1821521763 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b48136f40 con 0x7f9b480b1730 2024-03-25T18:19:48.714 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:48.713+0000 7f9b3e7fc640 1 -- 172.21.15.97:0/1821521763 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 184343783 0 0) 0x7f9b2c004eb0 con 0x7f9b4814be90 2024-03-25T18:19:48.714 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:48.713+0000 7f9b3e7fc640 1 -- 172.21.15.97:0/1821521763 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9b48143d90 con 0x7f9b4814be90 2024-03-25T18:19:48.714 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:48.713+0000 7f9b3e7fc640 1 -- 172.21.15.97:0/1821521763 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f9b40001c90 con 0x7f9b4814be90 2024-03-25T18:19:48.714 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:48.713+0000 7f9b3e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:48.714 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:48.713+0000 7f9b3e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:19:48.715 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:48.713+0000 7f9b3e7fc640 10 monclient(hunting): dump: 2024-03-25T18:19:48.715 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:19:48.715 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:19:48.715 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:19:48.715 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:19:48.715 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:48.715 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:19:48.715 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:19:48.715 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:19:48.715 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:19:48.715 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:19:48.715 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:48.713+0000 7f9b3e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:19:48.720 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:48.719+0000 7f9b3e7fc640 1 -- 172.21.15.97:0/1821521763 <== mon.1 v1:172.21.15.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3282476696 0 0) 0x7f9b48143d90 con 0x7f9b4814be90 2024-03-25T18:19:48.721 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:48.719+0000 7f9b3e7fc640 1 -- 172.21.15.97:0/1821521763 >> v1:172.21.15.97:6790/0 conn(0x7f9b480b1730 legacy=0x7f9b48137790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:48.721 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:48.719+0000 7f9b3e7fc640 1 -- 172.21.15.97:0/1821521763 >> v1:172.21.15.97:6789/0 conn(0x7f9b480b4d60 legacy=0x7f9b481daf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:48.721 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:48.719+0000 7f9b3e7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:19:48.721 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:48.719+0000 7f9b3e7fc640 1 monclient: found mon.b 2024-03-25T18:19:48.721 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:48.719+0000 7f9b3e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:48.721 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:48.719+0000 7f9b3e7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:19:48.721 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:48.719+0000 7f9b3e7fc640 1 -- 172.21.15.97:0/1821521763 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b48143f20 con 0x7f9b4814be90 2024-03-25T18:19:48.721 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:48.719+0000 7f9b3e7fc640 10 monclient: _finish_auth 0 2024-03-25T18:19:48.721 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:48.719+0000 7f9b3e7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:19:48.721 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:48.719+0000 7f9b3e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:48.721 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:48.719+0000 7f9b501af640 5 monclient: authenticate success, global_id 4378 2024-03-25T18:19:48.721 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:48.719+0000 7f9b501af640 10 monclient: _renew_subs 2024-03-25T18:19:48.721 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:48.719+0000 7f9b501af640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:19:48.721 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:48.719+0000 7f9b501af640 1 -- 172.21.15.97:0/1821521763 --> v1:172.21.15.98:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b481441d0 con 0x7f9b4814be90 2024-03-25T18:19:48.722 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:48.719+0000 7f9b501af640 10 client.4378.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:19:48.722 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:48.719+0000 7f9b501af640 10 monclient: _renew_subs 2024-03-25T18:19:48.722 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:48.719+0000 7f9b501af640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:19:48.722 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:48.719+0000 7f9b501af640 1 -- 172.21.15.97:0/1821521763 --> v1:172.21.15.98:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b481e3440 con 0x7f9b4814be90 2024-03-25T18:19:48.722 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:48.719+0000 7f9b501af640 1 librados: init done 2024-03-25T18:19:48.722 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:48.720+0000 7f9b501af640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:48.722 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:48.720+0000 7f9b501af640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:48.722 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:48.720+0000 7f9b501af640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:19:48.722 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:48.720+0000 7f9b501af640 1 -- 172.21.15.97:0/1821521763 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b481370d0 con 0x7f9b4814be90 2024-03-25T18:19:48.995 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:48.993+0000 7f9b3e7fc640 1 -- 172.21.15.97:0/1821521763 <== mon.1 v1:172.21.15.98:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9b40003990 con 0x7f9b4814be90 2024-03-25T18:19:48.995 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:48.993+0000 7f9b3e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:48.995 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:48.993+0000 7f9b3e7fc640 1 -- 172.21.15.97:0/1821521763 <== mon.1 v1:172.21.15.98:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f9b40003b40 con 0x7f9b4814be90 2024-03-25T18:19:48.995 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:48.993+0000 7f9b3e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:48.995 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:48.993+0000 7f9b3e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:19:48.995 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:48.993+0000 7f9b3e7fc640 10 monclient: dump: 2024-03-25T18:19:48.995 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:19:48.995 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:19:48.995 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:19:48.995 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:19:48.995 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:48.996 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:19:48.996 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:19:48.996 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:19:48.996 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:19:48.996 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:19:48.996 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:48.993+0000 7f9b3ffff640 4 set_mon_vals no callback set 2024-03-25T18:19:48.996 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:48.994+0000 7f9b3e7fc640 1 -- 172.21.15.97:0/1821521763 <== mon.1 v1:172.21.15.98:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2377622137 0 0) 0x7f9b40003d60 con 0x7f9b4814be90 2024-03-25T18:19:48.997 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:48.995+0000 7f9b4df4b640 10 monclient: get_auth_request con 0x7f9b2c06dda0 auth_method 0 2024-03-25T18:19:48.997 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:48.995+0000 7f9b3e7fc640 10 client.4378.objecter ms_handle_connect 0x7f9b2c06dda0 2024-03-25T18:19:49.001 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:48.999+0000 7f9b3e7fc640 1 -- 172.21.15.97:0/1821521763 <== mon.1 v1:172.21.15.98:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 430711918 0 0) 0x7f9b40054840 con 0x7f9b4814be90 2024-03-25T18:19:49.001 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:49.000+0000 7f9b3e7fc640 10 client.4378.objecter ms_dispatch 0x7f9b48000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:19:49.001 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:49.000+0000 7f9b3e7fc640 3 client.4378.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:19:49.001 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:49.000+0000 7f9b3e7fc640 3 client.4378.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:19:49.001 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:49.000+0000 7f9b3e7fc640 20 client.4378.objecter dump_active .. 0 homeless 2024-03-25T18:19:49.001 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:49.000+0000 7f9b3e7fc640 1 -- 172.21.15.97:0/1821521763 <== mon.1 v1:172.21.15.98:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9b40050350 con 0x7f9b4814be90 2024-03-25T18:19:49.001 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:49.000+0000 7f9b3e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:49.001 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:49.000+0000 7f9b3e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:19:49.238 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:49.236+0000 7f9b501af640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:49.238 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:49.236+0000 7f9b501af640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:49.238 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:49.236+0000 7f9b501af640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:19:49.238 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:49.236+0000 7f9b501af640 1 -- 172.21.15.97:0/1821521763 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9b481374b0 con 0x7f9b4814be90 2024-03-25T18:19:49.262 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:49.261+0000 7f9b3e7fc640 1 -- 172.21.15.97:0/1821521763 <== mon.1 v1:172.21.15.98:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2169427412) 0x7f9b481374b0 con 0x7f9b4814be90 2024-03-25T18:19:49.262 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:49.261+0000 7f9b3e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:49.262 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:49.261+0000 7f9b3e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:19:49.263 INFO:teuthology.misc.health.smithi097.stdout: 2024-03-25T18:19:49.263 INFO:teuthology.misc.health.smithi097.stdout:{"epoch":10,"fsid":"4c3d472c-79b8-486b-9e3d-2738d4306754","created":"2024-03-25T18:16:14.737309+0000","modified":"2024-03-25T18:16:20.357492+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:20.357492+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cb5152b9-262f-45c3-94b8-18d0e81df558","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6daf8793-0838-473d-a7d6-9a0d2894c75d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"136f1493-3b9c-4f43-a2e7-641482d69dae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"397d538c-4f64-4daf-9e8b-662d09f91327","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4f1df08-663f-4bda-93f8-3878ab7c24b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c4b18d70-a3ce-4b15-9b70-eae5525bd761","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8dd4033a-de0e-4f8a-96dc-59f3c1f825b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0077722e-ffa6-4a3d-9891-929d9869b286","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:19:49.265 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:49.263+0000 7f9b501af640 10 librados: watch_flush enter 2024-03-25T18:19:49.265 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:49.263+0000 7f9b501af640 10 librados: watch_flush exit 2024-03-25T18:19:49.265 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:49.263+0000 7f9b501af640 20 client.4378.objecter shutdown clearing up homeless session... 2024-03-25T18:19:49.266 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:49.263+0000 7f9b501af640 10 client.4378.objecter successfully canceled tick 2024-03-25T18:19:49.266 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:49.264+0000 7f9b501af640 1 -- 172.21.15.97:0/1821521763 >> v1:172.21.15.98:6800/800387573 conn(0x7f9b2c06dda0 legacy=0x7f9b2c070260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:49.266 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:49.264+0000 7f9b501af640 10 monclient: shutdown 2024-03-25T18:19:49.266 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:49.264+0000 7f9b501af640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:49.266 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:49.264+0000 7f9b501af640 1 -- 172.21.15.97:0/1821521763 >> v1:172.21.15.98:6789/0 conn(0x7f9b4814be90 legacy=0x7f9b481db620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:49.266 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:49.264+0000 7f9b501af640 1 -- 172.21.15.97:0/1821521763 shutdown_connections 2024-03-25T18:19:49.266 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:49.264+0000 7f9b501af640 1 -- 172.21.15.97:0/1821521763 >> 172.21.15.97:0/1821521763 conn(0x7f9b480617a0 msgr2=0x7f9b480b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:49.266 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:49.264+0000 7f9b501af640 1 -- 172.21.15.97:0/1821521763 shutdown_connections 2024-03-25T18:19:49.266 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:49.264+0000 7f9b501af640 1 -- 172.21.15.97:0/1821521763 wait complete. 2024-03-25T18:19:49.266 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:49.264+0000 7f9b501af640 1 librados: shutdown 2024-03-25T18:19:49.276 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:19:52.457 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:03:54.496 34931== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:52.516 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:03:54.555 34931== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:52.546 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:03:54.585 34931== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:52.581 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:03:54.625 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:52.585 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:03:54.624 34935== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:52.595 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:03:54.634 34935== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:52.595 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:03:54.639 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:52.624 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:03:54.668 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:52.632 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:03:54.671 34935== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:56.083 DEBUG:teuthology.orchestra.run.smithi097:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:19:56.195 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.194+0000 7feda54f6640 10 monclient: get_monmap_and_config 2024-03-25T18:19:56.196 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.194+0000 7feda54f6640 10 monclient: build_initial_monmap 2024-03-25T18:19:56.196 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.194+0000 7feda54f6640 10 monclient: monmap: 2024-03-25T18:19:56.196 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:19:56.196 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:19:56.196 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:19:56.195918+0000 2024-03-25T18:19:56.196 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:19:56.195918+0000 2024-03-25T18:19:56.196 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:56.196 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:19:56.196 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:19:56.196 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T18:19:56.196 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-b 2024-03-25T18:19:56.196 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:19:56.198 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.195+0000 7feda54f6640 1 Processor -- start 2024-03-25T18:19:56.198 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.195+0000 7feda54f6640 1 -- start start 2024-03-25T18:19:56.198 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.195+0000 7feda54f6640 10 monclient: init 2024-03-25T18:19:56.198 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.196+0000 7feda54f6640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:56.198 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.196+0000 7feda54f6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:19:56.198 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.196+0000 7feda54f6640 10 monclient(hunting): picked mon.noname-b con 0x7feda006ab80 addr v1:172.21.15.97:6790/0 2024-03-25T18:19:56.198 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.196+0000 7feda54f6640 10 monclient(hunting): picked mon.noname-c con 0x7feda006af60 addr v1:172.21.15.98:6789/0 2024-03-25T18:19:56.198 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.196+0000 7feda54f6640 10 monclient(hunting): picked mon.noname-a con 0x7feda00b00b0 addr v1:172.21.15.97:6789/0 2024-03-25T18:19:56.198 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.196+0000 7feda54f6640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feda00b0500 con 0x7feda00b00b0 2024-03-25T18:19:56.198 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.196+0000 7feda54f6640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feda013ffa0 con 0x7feda006af60 2024-03-25T18:19:56.198 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.196+0000 7feda54f6640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feda00af520 con 0x7feda006ab80 2024-03-25T18:19:56.199 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.196+0000 7feda54f6640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:56.199 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.196+0000 7feda54f6640 10 monclient(hunting): authenticate will time out at 7808.472593s 2024-03-25T18:19:56.210 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.209+0000 7fed97fff640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7feda006af60 0x7feda013f890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.97:59610/0 (socket says 172.21.15.97:59610) 2024-03-25T18:19:56.210 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.209+0000 7fed97fff640 1 -- 172.21.15.97:0/1325158180 learned_addr learned my addr 172.21.15.97:0/1325158180 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:19:56.222 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.221+0000 7fed9e7fc640 1 -- 172.21.15.97:0/1325158180 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 628756488 0 0) 0x7feda00af520 con 0x7feda006ab80 2024-03-25T18:19:56.222 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.221+0000 7fed9e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:19:56.223 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.221+0000 7fed9e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:56.223 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.221+0000 7fed9e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:56.223 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.221+0000 7fed9e7fc640 10 monclient(hunting): my global_id is 4376 2024-03-25T18:19:56.223 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.221+0000 7fed9e7fc640 1 -- 172.21.15.97:0/1325158180 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed8c003520 con 0x7feda006ab80 2024-03-25T18:19:56.226 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.225+0000 7fed9e7fc640 1 -- 172.21.15.97:0/1325158180 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2223505376 0 0) 0x7feda013ffa0 con 0x7feda006af60 2024-03-25T18:19:56.226 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.225+0000 7fed9e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:19:56.226 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.225+0000 7fed9e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:56.226 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.225+0000 7fed9e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:56.226 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.225+0000 7fed9e7fc640 10 monclient(hunting): my global_id is 4381 2024-03-25T18:19:56.226 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.225+0000 7fed9e7fc640 1 -- 172.21.15.97:0/1325158180 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feda00af520 con 0x7feda006af60 2024-03-25T18:19:56.235 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.234+0000 7fed9e7fc640 1 -- 172.21.15.97:0/1325158180 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1950038569 0 0) 0x7feda00b0500 con 0x7feda00b00b0 2024-03-25T18:19:56.235 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.234+0000 7fed9e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:19:56.235 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.234+0000 7fed9e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:56.235 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.234+0000 7fed9e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:56.235 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.234+0000 7fed9e7fc640 10 monclient(hunting): my global_id is 4383 2024-03-25T18:19:56.235 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.234+0000 7fed9e7fc640 1 -- 172.21.15.97:0/1325158180 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feda013ffa0 con 0x7feda00b00b0 2024-03-25T18:19:56.237 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.235+0000 7fed9e7fc640 1 -- 172.21.15.97:0/1325158180 <== mon.2 v1:172.21.15.97:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 435880883 0 0) 0x7fed8c003520 con 0x7feda006ab80 2024-03-25T18:19:56.237 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.235+0000 7fed9e7fc640 1 -- 172.21.15.97:0/1325158180 >> v1:172.21.15.98:6789/0 conn(0x7feda006af60 legacy=0x7feda013f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:56.237 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.235+0000 7fed9e7fc640 1 -- 172.21.15.97:0/1325158180 >> v1:172.21.15.97:6789/0 conn(0x7feda00b00b0 legacy=0x7feda014dcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:56.237 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.235+0000 7fed9e7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:19:56.237 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.235+0000 7fed9e7fc640 1 monclient: found mon.noname-b 2024-03-25T18:19:56.237 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.235+0000 7fed9e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:56.238 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.235+0000 7fed9e7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.97:6790/0 2024-03-25T18:19:56.238 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.235+0000 7fed9e7fc640 1 -- 172.21.15.97:0/1325158180 --> v1:172.21.15.97:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feda0072890 con 0x7feda006ab80 2024-03-25T18:19:56.238 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.235+0000 7fed9e7fc640 10 monclient: _finish_auth 0 2024-03-25T18:19:56.238 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.235+0000 7fed9e7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:19:56.238 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.235+0000 7fed9e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:56.238 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.235+0000 7fed9e7fc640 1 -- 172.21.15.97:0/1325158180 <== mon.2 v1:172.21.15.97:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7fed90001690 con 0x7feda006ab80 2024-03-25T18:19:56.238 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.235+0000 7fed9e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:56.238 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.235+0000 7fed9e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:19:56.238 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.235+0000 7fed9e7fc640 10 monclient: dump: 2024-03-25T18:19:56.238 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:19:56.238 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:19:56.238 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:19:56.238 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:19:56.238 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:56.239 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:19:56.239 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:19:56.239 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:19:56.239 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:19:56.239 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:19:56.239 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.235+0000 7feda54f6640 5 monclient: authenticate success, global_id 4376 2024-03-25T18:19:56.239 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.235+0000 7feda54f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:56.248 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.246+0000 7fed9e7fc640 1 -- 172.21.15.97:0/1325158180 <== mon.2 v1:172.21.15.97:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fed900017f0 con 0x7feda006ab80 2024-03-25T18:19:56.248 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.247+0000 7fed9e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:56.248 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.247+0000 7fed9e7fc640 1 -- 172.21.15.97:0/1325158180 <== mon.2 v1:172.21.15.97:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7fed90001c00 con 0x7feda006ab80 2024-03-25T18:19:56.248 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.247+0000 7fed9e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:56.248 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.247+0000 7fed9e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:19:56.248 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.247+0000 7fed9e7fc640 10 monclient: dump: 2024-03-25T18:19:56.248 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:19:56.248 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:19:56.248 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:19:56.248 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:19:56.248 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:56.249 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:19:56.249 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:19:56.249 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:19:56.249 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:19:56.249 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:19:56.249 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.247+0000 7feda54f6640 10 monclient: get_monmap_and_config success 2024-03-25T18:19:56.249 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.247+0000 7feda54f6640 4 set_mon_vals no callback set 2024-03-25T18:19:56.251 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.247+0000 7feda54f6640 10 monclient: shutdown 2024-03-25T18:19:56.251 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.247+0000 7feda54f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:56.251 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.247+0000 7feda54f6640 1 -- 172.21.15.97:0/1325158180 >> v1:172.21.15.97:6790/0 conn(0x7feda006ab80 legacy=0x7feda00ad880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:56.251 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.248+0000 7feda54f6640 1 -- 172.21.15.97:0/1325158180 shutdown_connections 2024-03-25T18:19:56.251 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.248+0000 7feda54f6640 1 -- 172.21.15.97:0/1325158180 >> 172.21.15.97:0/1325158180 conn(0x7feda00b4a80 msgr2=0x7feda00b4e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:56.251 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.248+0000 7feda54f6640 1 -- 172.21.15.97:0/1325158180 shutdown_connections 2024-03-25T18:19:56.251 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.248+0000 7feda54f6640 1 -- 172.21.15.97:0/1325158180 wait complete. 2024-03-25T18:19:56.251 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.248+0000 7feda54f6640 10 monclient: build_initial_monmap 2024-03-25T18:19:56.251 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.248+0000 7feda54f6640 10 monclient: monmap: 2024-03-25T18:19:56.252 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:19:56.252 INFO:teuthology.misc.health.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:19:56.252 INFO:teuthology.misc.health.smithi097.stderr:last_changed 0.000000 2024-03-25T18:19:56.252 INFO:teuthology.misc.health.smithi097.stderr:created 0.000000 2024-03-25T18:19:56.252 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:56.252 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 1 2024-03-25T18:19:56.252 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:19:56.252 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T18:19:56.252 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-c 2024-03-25T18:19:56.252 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:19:56.252 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.248+0000 7feda54f6640 1 librados: starting msgr at 2024-03-25T18:19:56.252 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.248+0000 7feda54f6640 1 librados: starting objecter 2024-03-25T18:19:56.252 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.249+0000 7feda54f6640 1 Processor -- start 2024-03-25T18:19:56.252 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.249+0000 7feda54f6640 1 -- start start 2024-03-25T18:19:56.252 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.249+0000 7feda54f6640 1 librados: setting wanted keys 2024-03-25T18:19:56.252 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.249+0000 7feda54f6640 1 librados: calling monclient init 2024-03-25T18:19:56.253 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.249+0000 7feda54f6640 10 monclient: init 2024-03-25T18:19:56.253 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.249+0000 7feda54f6640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:56.253 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.249+0000 7feda54f6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:19:56.253 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.249+0000 7feda54f6640 10 monclient(hunting): picked mon.noname-a con 0x7feda006ab80 addr v1:172.21.15.97:6789/0 2024-03-25T18:19:56.253 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.249+0000 7feda54f6640 10 monclient(hunting): picked mon.noname-b con 0x7feda00b00b0 addr v1:172.21.15.98:6789/0 2024-03-25T18:19:56.253 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.249+0000 7feda54f6640 10 monclient(hunting): picked mon.noname-c con 0x7feda01d7f70 addr v1:172.21.15.97:6790/0 2024-03-25T18:19:56.253 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.249+0000 7feda54f6640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feda0145590 con 0x7feda006ab80 2024-03-25T18:19:56.253 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.249+0000 7feda54f6640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feda0145ae0 con 0x7feda00b00b0 2024-03-25T18:19:56.253 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.249+0000 7feda54f6640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feda01466d0 con 0x7feda01d7f70 2024-03-25T18:19:56.253 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.249+0000 7feda54f6640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:56.253 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.249+0000 7feda54f6640 10 monclient(hunting): authenticate will time out at 7808.525758s 2024-03-25T18:19:56.264 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.263+0000 7fed9f7fe640 1 --1- >> v1:172.21.15.97:6790/0 conn(0x7feda01d7f70 0x7feda01e2ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.97:6790/0 says I am v1:172.21.15.97:42760/0 (socket says 172.21.15.97:42760) 2024-03-25T18:19:56.264 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.263+0000 7fed9f7fe640 1 -- 172.21.15.97:0/3871222634 learned_addr learned my addr 172.21.15.97:0/3871222634 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:19:56.265 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.264+0000 7fed977fe640 10 client.?.objecter ms_handle_connect 0x7feda01d7f70 2024-03-25T18:19:56.265 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.264+0000 7fed977fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:56.266 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.265+0000 7fed977fe640 10 client.?.objecter ms_handle_connect 0x7feda006ab80 2024-03-25T18:19:56.266 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.265+0000 7fed977fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:56.270 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.268+0000 7fed977fe640 1 -- 172.21.15.97:0/3871222634 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1315664338 0 0) 0x7feda0145590 con 0x7feda006ab80 2024-03-25T18:19:56.270 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.268+0000 7fed977fe640 10 monclient(hunting): _negotiate 2024-03-25T18:19:56.270 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.268+0000 7fed977fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:56.270 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.268+0000 7fed977fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:56.270 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.268+0000 7fed977fe640 10 monclient(hunting): my global_id is 4386 2024-03-25T18:19:56.270 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.268+0000 7fed977fe640 1 -- 172.21.15.97:0/3871222634 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed80004eb0 con 0x7feda006ab80 2024-03-25T18:19:56.271 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.270+0000 7fed977fe640 10 client.?.objecter ms_handle_connect 0x7feda00b00b0 2024-03-25T18:19:56.271 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.270+0000 7fed977fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:56.278 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.276+0000 7fed977fe640 1 -- 172.21.15.97:0/3871222634 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3450976167 0 0) 0x7feda0145ae0 con 0x7feda00b00b0 2024-03-25T18:19:56.278 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.276+0000 7fed977fe640 10 monclient(hunting): _negotiate 2024-03-25T18:19:56.278 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.276+0000 7fed977fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:56.278 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.276+0000 7fed977fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:56.278 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.276+0000 7fed977fe640 10 monclient(hunting): my global_id is 4384 2024-03-25T18:19:56.278 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.276+0000 7fed977fe640 1 -- 172.21.15.97:0/3871222634 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feda0145590 con 0x7feda00b00b0 2024-03-25T18:19:56.281 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.279+0000 7fed977fe640 1 -- 172.21.15.97:0/3871222634 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1313151395 0 0) 0x7feda01466d0 con 0x7feda01d7f70 2024-03-25T18:19:56.281 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.279+0000 7fed977fe640 10 monclient(hunting): _negotiate 2024-03-25T18:19:56.281 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.279+0000 7fed977fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:56.281 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.279+0000 7fed977fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:56.281 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.279+0000 7fed977fe640 10 monclient(hunting): my global_id is 4379 2024-03-25T18:19:56.281 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.279+0000 7fed977fe640 1 -- 172.21.15.97:0/3871222634 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feda0145ae0 con 0x7feda01d7f70 2024-03-25T18:19:56.282 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.279+0000 7fed977fe640 1 -- 172.21.15.97:0/3871222634 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2913622545 0 0) 0x7fed80004eb0 con 0x7feda006ab80 2024-03-25T18:19:56.282 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.279+0000 7fed977fe640 1 -- 172.21.15.97:0/3871222634 --> v1:172.21.15.97:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feda01466d0 con 0x7feda006ab80 2024-03-25T18:19:56.282 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.279+0000 7fed977fe640 1 -- 172.21.15.97:0/3871222634 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7fed90001b00 con 0x7feda006ab80 2024-03-25T18:19:56.282 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.279+0000 7fed977fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:56.282 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.279+0000 7fed977fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:19:56.282 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.279+0000 7fed977fe640 10 monclient(hunting): dump: 2024-03-25T18:19:56.282 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:19:56.282 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:19:56.282 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:19:56.282 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:19:56.282 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:56.282 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:19:56.282 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:19:56.283 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:19:56.283 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:19:56.283 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:19:56.283 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.279+0000 7fed977fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:19:56.288 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.287+0000 7fed977fe640 1 -- 172.21.15.97:0/3871222634 <== mon.0 v1:172.21.15.97:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 392529826 0 0) 0x7feda01466d0 con 0x7feda006ab80 2024-03-25T18:19:56.288 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.287+0000 7fed977fe640 1 -- 172.21.15.97:0/3871222634 >> v1:172.21.15.97:6790/0 conn(0x7feda01d7f70 legacy=0x7feda01e2ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:56.288 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.287+0000 7fed977fe640 1 -- 172.21.15.97:0/3871222634 >> v1:172.21.15.98:6789/0 conn(0x7feda00b00b0 legacy=0x7feda01d7860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:56.289 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.287+0000 7fed977fe640 10 monclient: _finish_hunting 0 2024-03-25T18:19:56.289 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.287+0000 7fed977fe640 1 monclient: found mon.a 2024-03-25T18:19:56.289 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.287+0000 7fed977fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:56.289 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.287+0000 7fed977fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:19:56.289 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.287+0000 7fed977fe640 1 -- 172.21.15.97:0/3871222634 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feda0144170 con 0x7feda006ab80 2024-03-25T18:19:56.289 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.287+0000 7fed977fe640 10 monclient: _finish_auth 0 2024-03-25T18:19:56.289 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.287+0000 7fed977fe640 10 monclient: _check_auth_tickets 2024-03-25T18:19:56.289 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.287+0000 7fed977fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:56.289 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.287+0000 7feda54f6640 5 monclient: authenticate success, global_id 4386 2024-03-25T18:19:56.289 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.287+0000 7feda54f6640 10 monclient: _renew_subs 2024-03-25T18:19:56.289 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.287+0000 7feda54f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:19:56.289 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.287+0000 7feda54f6640 1 -- 172.21.15.97:0/3871222634 --> v1:172.21.15.97:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feda01443c0 con 0x7feda006ab80 2024-03-25T18:19:56.289 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.287+0000 7feda54f6640 10 client.4386.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:19:56.289 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.287+0000 7feda54f6640 10 monclient: _renew_subs 2024-03-25T18:19:56.290 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.287+0000 7feda54f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:19:56.290 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.287+0000 7feda54f6640 1 -- 172.21.15.97:0/3871222634 --> v1:172.21.15.97:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7feda01d7460 con 0x7feda006ab80 2024-03-25T18:19:56.290 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.287+0000 7feda54f6640 1 librados: init done 2024-03-25T18:19:56.290 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.288+0000 7feda54f6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:56.290 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.288+0000 7feda54f6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:56.290 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.288+0000 7feda54f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:19:56.290 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.288+0000 7feda54f6640 1 -- 172.21.15.97:0/3871222634 --> v1:172.21.15.97:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feda0145940 con 0x7feda006ab80 2024-03-25T18:19:56.593 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.592+0000 7fed977fe640 1 -- 172.21.15.97:0/3871222634 <== mon.0 v1:172.21.15.97:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fed90001220 con 0x7feda006ab80 2024-03-25T18:19:56.594 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.592+0000 7fed977fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:56.594 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.592+0000 7fed977fe640 1 -- 172.21.15.97:0/3871222634 <== mon.0 v1:172.21.15.97:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7fed900038a0 con 0x7feda006ab80 2024-03-25T18:19:56.594 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.592+0000 7fed977fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:56.594 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.592+0000 7fed977fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:19:56.594 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.592+0000 7fed977fe640 10 monclient: dump: 2024-03-25T18:19:56.594 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:19:56.594 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:19:56.594 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:19:56.594 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:19:56.594 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:56.594 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:19:56.594 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:19:56.595 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:19:56.595 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:19:56.595 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:19:56.595 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.592+0000 7fed9dffb640 4 set_mon_vals no callback set 2024-03-25T18:19:56.595 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.593+0000 7fed977fe640 1 -- 172.21.15.97:0/3871222634 <== mon.0 v1:172.21.15.97:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2377622137 0 0) 0x7fed90019870 con 0x7feda006ab80 2024-03-25T18:19:56.595 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.593+0000 7fed97fff640 10 monclient: get_auth_request con 0x7fed8006de00 auth_method 0 2024-03-25T18:19:56.595 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.594+0000 7fed977fe640 10 client.4386.objecter ms_handle_connect 0x7fed8006de00 2024-03-25T18:19:56.599 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.598+0000 7fed977fe640 1 -- 172.21.15.97:0/3871222634 <== mon.0 v1:172.21.15.97:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 430711918 0 0) 0x7fed90053500 con 0x7feda006ab80 2024-03-25T18:19:56.599 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.598+0000 7fed977fe640 10 client.4386.objecter ms_dispatch 0x7feda0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:19:56.599 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.598+0000 7fed977fe640 3 client.4386.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:19:56.599 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.598+0000 7fed977fe640 3 client.4386.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:19:56.599 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.598+0000 7fed977fe640 20 client.4386.objecter dump_active .. 0 homeless 2024-03-25T18:19:56.599 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.598+0000 7fed977fe640 1 -- 172.21.15.97:0/3871222634 <== mon.0 v1:172.21.15.97:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fed9004ee40 con 0x7feda006ab80 2024-03-25T18:19:56.599 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.598+0000 7fed977fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:56.599 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.598+0000 7fed977fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:19:56.825 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.823+0000 7feda54f6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:56.825 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.823+0000 7feda54f6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:56.825 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.823+0000 7feda54f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:19:56.825 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.823+0000 7feda54f6640 1 -- 172.21.15.97:0/3871222634 --> v1:172.21.15.97:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7feda0145cc0 con 0x7feda006ab80 2024-03-25T18:19:56.855 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.854+0000 7fed977fe640 1 -- 172.21.15.97:0/3871222634 <== mon.0 v1:172.21.15.97:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2169427412) 0x7fed90050f40 con 0x7feda006ab80 2024-03-25T18:19:56.855 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.854+0000 7fed977fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:56.855 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.854+0000 7fed977fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:19:56.855 INFO:teuthology.misc.health.smithi097.stdout: 2024-03-25T18:19:56.856 INFO:teuthology.misc.health.smithi097.stdout:{"epoch":10,"fsid":"4c3d472c-79b8-486b-9e3d-2738d4306754","created":"2024-03-25T18:16:14.737309+0000","modified":"2024-03-25T18:16:20.357492+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:20.357492+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cb5152b9-262f-45c3-94b8-18d0e81df558","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6daf8793-0838-473d-a7d6-9a0d2894c75d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"136f1493-3b9c-4f43-a2e7-641482d69dae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"397d538c-4f64-4daf-9e8b-662d09f91327","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4f1df08-663f-4bda-93f8-3878ab7c24b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c4b18d70-a3ce-4b15-9b70-eae5525bd761","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8dd4033a-de0e-4f8a-96dc-59f3c1f825b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0077722e-ffa6-4a3d-9891-929d9869b286","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:19:56.858 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.856+0000 7feda54f6640 10 librados: watch_flush enter 2024-03-25T18:19:56.858 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.856+0000 7feda54f6640 10 librados: watch_flush exit 2024-03-25T18:19:56.858 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.856+0000 7feda54f6640 20 client.4386.objecter shutdown clearing up homeless session... 2024-03-25T18:19:56.859 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.856+0000 7feda54f6640 10 client.4386.objecter successfully canceled tick 2024-03-25T18:19:56.859 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.856+0000 7feda54f6640 1 -- 172.21.15.97:0/3871222634 >> v1:172.21.15.98:6800/800387573 conn(0x7fed8006de00 legacy=0x7fed800702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:56.859 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.856+0000 7feda54f6640 10 monclient: shutdown 2024-03-25T18:19:56.859 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.856+0000 7feda54f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:56.859 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.856+0000 7feda54f6640 1 -- 172.21.15.97:0/3871222634 >> v1:172.21.15.97:6789/0 conn(0x7feda006ab80 legacy=0x7feda0145fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:56.859 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.856+0000 7feda54f6640 1 -- 172.21.15.97:0/3871222634 shutdown_connections 2024-03-25T18:19:56.859 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.856+0000 7feda54f6640 1 -- 172.21.15.97:0/3871222634 >> 172.21.15.97:0/3871222634 conn(0x7feda00b4a80 msgr2=0x7feda00b4e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:56.859 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.857+0000 7feda54f6640 1 -- 172.21.15.97:0/3871222634 shutdown_connections 2024-03-25T18:19:56.859 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.857+0000 7feda54f6640 1 -- 172.21.15.97:0/3871222634 wait complete. 2024-03-25T18:19:56.859 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:19:56.857+0000 7feda54f6640 1 librados: shutdown 2024-03-25T18:19:56.869 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:20:03.676 DEBUG:teuthology.orchestra.run.smithi097:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:20:03.788 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:03.786+0000 7f7274997640 10 monclient: get_monmap_and_config 2024-03-25T18:20:03.788 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:03.786+0000 7f7274997640 10 monclient: build_initial_monmap 2024-03-25T18:20:03.788 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:03.786+0000 7f7274997640 10 monclient: monmap: 2024-03-25T18:20:03.788 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:20:03.788 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:20:03.788 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:20:03.787937+0000 2024-03-25T18:20:03.788 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:20:03.787937+0000 2024-03-25T18:20:03.788 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:03.788 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:20:03.788 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:20:03.789 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T18:20:03.789 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-b 2024-03-25T18:20:03.789 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:20:03.790 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:03.787+0000 7f7274997640 1 Processor -- start 2024-03-25T18:20:03.790 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:03.787+0000 7f7274997640 1 -- start start 2024-03-25T18:20:03.790 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:03.787+0000 7f7274997640 10 monclient: init 2024-03-25T18:20:03.790 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:03.788+0000 7f7274997640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:03.790 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:03.788+0000 7f7274997640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:20:03.790 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:03.788+0000 7f7274997640 10 monclient(hunting): picked mon.noname-b con 0x7f72700a98f0 addr v1:172.21.15.97:6790/0 2024-03-25T18:20:03.790 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:03.788+0000 7f7274997640 10 monclient(hunting): picked mon.noname-a con 0x7f727013f710 addr v1:172.21.15.97:6789/0 2024-03-25T18:20:03.790 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:03.788+0000 7f7274997640 10 monclient(hunting): picked mon.noname-c con 0x7f72701430c0 addr v1:172.21.15.98:6789/0 2024-03-25T18:20:03.790 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:03.788+0000 7f7274997640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7270142520 con 0x7f727013f710 2024-03-25T18:20:03.790 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:03.788+0000 7f7274997640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72700b28a0 con 0x7f72701430c0 2024-03-25T18:20:03.790 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:03.788+0000 7f7274997640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72700b2a30 con 0x7f72700a98f0 2024-03-25T18:20:03.790 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:03.788+0000 7f7274997640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:03.790 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:03.788+0000 7f7274997640 10 monclient(hunting): authenticate will time out at 7816.064086s 2024-03-25T18:20:03.803 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:03.802+0000 7f726ed9d640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f72701430c0 0x7f727014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.97:60090/0 (socket says 172.21.15.97:60090) 2024-03-25T18:20:03.803 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:03.802+0000 7f726ed9d640 1 -- 172.21.15.97:0/2242187582 learned_addr learned my addr 172.21.15.97:0/2242187582 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:20:03.815 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:03.814+0000 7f726dd9b640 1 -- 172.21.15.97:0/2242187582 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2517285673 0 0) 0x7f72700b28a0 con 0x7f72701430c0 2024-03-25T18:20:03.815 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:03.814+0000 7f726dd9b640 10 monclient(hunting): _negotiate 2024-03-25T18:20:03.815 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:03.814+0000 7f726dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:03.816 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:03.814+0000 7f726dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:03.816 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:03.814+0000 7f726dd9b640 10 monclient(hunting): my global_id is 4387 2024-03-25T18:20:03.816 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:03.814+0000 7f726dd9b640 1 -- 172.21.15.97:0/2242187582 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f725c003520 con 0x7f72701430c0 2024-03-25T18:20:03.822 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:03.821+0000 7f726dd9b640 1 -- 172.21.15.97:0/2242187582 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 407396117 0 0) 0x7f72700b2a30 con 0x7f72700a98f0 2024-03-25T18:20:03.822 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:03.821+0000 7f726dd9b640 10 monclient(hunting): _negotiate 2024-03-25T18:20:03.822 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:03.821+0000 7f726dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:03.822 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:03.821+0000 7f726dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:03.822 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:03.821+0000 7f726dd9b640 10 monclient(hunting): my global_id is 4382 2024-03-25T18:20:03.823 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:03.821+0000 7f726dd9b640 1 -- 172.21.15.97:0/2242187582 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72700b28a0 con 0x7f72700a98f0 2024-03-25T18:20:03.827 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:03.826+0000 7f726dd9b640 1 -- 172.21.15.97:0/2242187582 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 981632824 0 0) 0x7f725c003520 con 0x7f72701430c0 2024-03-25T18:20:03.827 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:03.826+0000 7f726dd9b640 1 -- 172.21.15.97:0/2242187582 >> v1:172.21.15.97:6790/0 conn(0x7f72700a98f0 legacy=0x7f72700a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:03.827 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:03.826+0000 7f726dd9b640 1 -- 172.21.15.97:0/2242187582 >> v1:172.21.15.97:6789/0 conn(0x7f727013f710 legacy=0x7f72701429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:03.828 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:03.826+0000 7f726dd9b640 10 monclient: _finish_hunting 0 2024-03-25T18:20:03.828 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:03.826+0000 7f726dd9b640 1 monclient: found mon.noname-c 2024-03-25T18:20:03.828 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:03.826+0000 7f726dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:03.828 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:03.826+0000 7f726dd9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.98:6789/0 2024-03-25T18:20:03.828 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:03.826+0000 7f726dd9b640 1 -- 172.21.15.97:0/2242187582 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72700b44c0 con 0x7f72701430c0 2024-03-25T18:20:03.828 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:03.826+0000 7f726dd9b640 10 monclient: _finish_auth 0 2024-03-25T18:20:03.828 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:03.826+0000 7f726dd9b640 10 monclient: _check_auth_tickets 2024-03-25T18:20:03.828 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:03.826+0000 7f726dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:03.828 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:03.826+0000 7f726dd9b640 1 -- 172.21.15.97:0/2242187582 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f7268001a10 con 0x7f72701430c0 2024-03-25T18:20:03.828 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:03.826+0000 7f726dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:03.828 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:03.826+0000 7f726dd9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:20:03.828 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:03.826+0000 7f726dd9b640 10 monclient: dump: 2024-03-25T18:20:03.828 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:20:03.828 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:20:03.828 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:20:03.829 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:20:03.829 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:03.829 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:20:03.829 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:20:03.829 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:20:03.829 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:20:03.829 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:20:03.829 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:03.826+0000 7f7274997640 5 monclient: authenticate success, global_id 4387 2024-03-25T18:20:03.829 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:03.826+0000 7f7274997640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:20:03.832 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:03.831+0000 7f726dd9b640 1 -- 172.21.15.97:0/2242187582 <== mon.1 v1:172.21.15.98:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7268001b70 con 0x7f72701430c0 2024-03-25T18:20:03.833 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:03.831+0000 7f726dd9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:03.833 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:03.831+0000 7f726dd9b640 1 -- 172.21.15.97:0/2242187582 <== mon.1 v1:172.21.15.98:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f7268001f80 con 0x7f72701430c0 2024-03-25T18:20:03.833 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:03.831+0000 7f726dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:03.833 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:03.831+0000 7f726dd9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:20:03.833 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:03.831+0000 7f726dd9b640 10 monclient: dump: 2024-03-25T18:20:03.833 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:20:03.833 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:20:03.833 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:20:03.833 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:20:03.833 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:03.833 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:20:03.833 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:20:03.833 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:20:03.834 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:20:03.834 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:20:03.834 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:03.831+0000 7f7274997640 10 monclient: get_monmap_and_config success 2024-03-25T18:20:03.834 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:03.831+0000 7f7274997640 4 set_mon_vals no callback set 2024-03-25T18:20:03.835 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:03.832+0000 7f7274997640 10 monclient: shutdown 2024-03-25T18:20:03.835 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:03.832+0000 7f7274997640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:03.835 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:03.832+0000 7f7274997640 1 -- 172.21.15.97:0/2242187582 >> v1:172.21.15.98:6789/0 conn(0x7f72701430c0 legacy=0x7f727014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:03.835 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:03.832+0000 7f7274997640 1 -- 172.21.15.97:0/2242187582 shutdown_connections 2024-03-25T18:20:03.835 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:03.832+0000 7f7274997640 1 -- 172.21.15.97:0/2242187582 >> 172.21.15.97:0/2242187582 conn(0x7f72700b7ea0 msgr2=0x7f72700b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:03.835 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:03.832+0000 7f7274997640 1 -- 172.21.15.97:0/2242187582 shutdown_connections 2024-03-25T18:20:03.835 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:03.832+0000 7f7274997640 1 -- 172.21.15.97:0/2242187582 wait complete. 2024-03-25T18:20:03.835 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:03.833+0000 7f7274997640 10 monclient: build_initial_monmap 2024-03-25T18:20:03.835 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:03.833+0000 7f7274997640 10 monclient: monmap: 2024-03-25T18:20:03.835 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:20:03.835 INFO:teuthology.misc.health.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:20:03.835 INFO:teuthology.misc.health.smithi097.stderr:last_changed 0.000000 2024-03-25T18:20:03.835 INFO:teuthology.misc.health.smithi097.stderr:created 0.000000 2024-03-25T18:20:03.835 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:03.836 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 1 2024-03-25T18:20:03.836 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:20:03.836 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T18:20:03.836 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-c 2024-03-25T18:20:03.836 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:20:03.836 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:03.834+0000 7f7274997640 1 librados: starting msgr at 2024-03-25T18:20:03.836 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:03.834+0000 7f7274997640 1 librados: starting objecter 2024-03-25T18:20:03.836 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:03.834+0000 7f7274997640 1 Processor -- start 2024-03-25T18:20:03.837 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:03.834+0000 7f7274997640 1 -- start start 2024-03-25T18:20:03.837 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:03.834+0000 7f7274997640 1 librados: setting wanted keys 2024-03-25T18:20:03.837 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:03.834+0000 7f7274997640 1 librados: calling monclient init 2024-03-25T18:20:03.837 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:03.834+0000 7f7274997640 10 monclient: init 2024-03-25T18:20:03.837 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:03.835+0000 7f7274997640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:03.837 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:03.835+0000 7f7274997640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:20:03.837 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:03.835+0000 7f7274997640 10 monclient(hunting): picked mon.noname-b con 0x7f72700a98f0 addr v1:172.21.15.98:6789/0 2024-03-25T18:20:03.837 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:03.835+0000 7f7274997640 10 monclient(hunting): picked mon.noname-a con 0x7f727013f710 addr v1:172.21.15.97:6789/0 2024-03-25T18:20:03.837 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:03.835+0000 7f7274997640 10 monclient(hunting): picked mon.noname-c con 0x7f72701430c0 addr v1:172.21.15.97:6790/0 2024-03-25T18:20:03.837 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:03.835+0000 7f7274997640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72700b7a80 con 0x7f727013f710 2024-03-25T18:20:03.837 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:03.835+0000 7f7274997640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72700b77a0 con 0x7f72700a98f0 2024-03-25T18:20:03.837 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:03.835+0000 7f7274997640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7270136bb0 con 0x7f72701430c0 2024-03-25T18:20:03.838 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:03.835+0000 7f7274997640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:03.838 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:03.835+0000 7f7274997640 10 monclient(hunting): authenticate will time out at 7816.111158s 2024-03-25T18:20:03.842 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:03.840+0000 7f726e59c640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f72700a98f0 0x7f7270137590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.97:60140/0 (socket says 172.21.15.97:60140) 2024-03-25T18:20:03.842 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:03.840+0000 7f726e59c640 1 -- 172.21.15.97:0/1638013044 learned_addr learned my addr 172.21.15.97:0/1638013044 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:20:03.842 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:03.841+0000 7f7266ffd640 10 client.?.objecter ms_handle_connect 0x7f72700a98f0 2024-03-25T18:20:03.842 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:03.841+0000 7f7266ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:03.850 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:03.849+0000 7f7266ffd640 1 -- 172.21.15.97:0/1638013044 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 529312241 0 0) 0x7f72700b77a0 con 0x7f72700a98f0 2024-03-25T18:20:03.850 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:03.849+0000 7f7266ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:20:03.850 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:03.849+0000 7f7266ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:03.851 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:03.849+0000 7f7266ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:03.851 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:03.849+0000 7f7266ffd640 10 monclient(hunting): my global_id is 4390 2024-03-25T18:20:03.851 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:03.849+0000 7f7266ffd640 1 -- 172.21.15.97:0/1638013044 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7250004eb0 con 0x7f72700a98f0 2024-03-25T18:20:03.854 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:03.853+0000 7f7266ffd640 10 client.?.objecter ms_handle_connect 0x7f72701430c0 2024-03-25T18:20:03.854 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:03.853+0000 7f7266ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:03.857 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:03.856+0000 7f7266ffd640 10 client.?.objecter ms_handle_connect 0x7f727013f710 2024-03-25T18:20:03.857 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:03.856+0000 7f7266ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:03.857 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:03.856+0000 7f7266ffd640 1 -- 172.21.15.97:0/1638013044 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 505624761 0 0) 0x7f7250004eb0 con 0x7f72700a98f0 2024-03-25T18:20:03.857 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:03.856+0000 7f7266ffd640 1 -- 172.21.15.97:0/1638013044 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f72700b77a0 con 0x7f72700a98f0 2024-03-25T18:20:03.858 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:03.857+0000 7f7266ffd640 1 -- 172.21.15.97:0/1638013044 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f7260001af0 con 0x7f72700a98f0 2024-03-25T18:20:03.858 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:03.857+0000 7f7266ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:03.858 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:03.857+0000 7f7266ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:20:03.858 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:03.857+0000 7f7266ffd640 10 monclient(hunting): dump: 2024-03-25T18:20:03.858 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:20:03.858 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:20:03.858 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:20:03.859 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:20:03.859 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:03.859 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:20:03.859 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:20:03.859 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:20:03.859 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:20:03.859 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:20:03.859 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:03.857+0000 7f7266ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:20:03.861 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:03.860+0000 7f7266ffd640 1 -- 172.21.15.97:0/1638013044 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2859096545 0 0) 0x7f7270136bb0 con 0x7f72701430c0 2024-03-25T18:20:03.861 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:03.860+0000 7f7266ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:20:03.861 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:03.860+0000 7f7266ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:03.861 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:03.860+0000 7f7266ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:03.862 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:03.860+0000 7f7266ffd640 10 monclient(hunting): my global_id is 4385 2024-03-25T18:20:03.862 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:03.860+0000 7f7266ffd640 1 -- 172.21.15.97:0/1638013044 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7250004eb0 con 0x7f72701430c0 2024-03-25T18:20:03.863 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:03.862+0000 7f7266ffd640 1 -- 172.21.15.97:0/1638013044 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 654068141 0 0) 0x7f72700b7a80 con 0x7f727013f710 2024-03-25T18:20:03.863 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:03.862+0000 7f7266ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:20:03.863 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:03.862+0000 7f7266ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:03.863 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:03.862+0000 7f7266ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:03.863 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:03.862+0000 7f7266ffd640 10 monclient(hunting): my global_id is 4392 2024-03-25T18:20:03.863 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:03.862+0000 7f7266ffd640 1 -- 172.21.15.97:0/1638013044 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7270136bb0 con 0x7f727013f710 2024-03-25T18:20:03.872 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:03.870+0000 7f7266ffd640 1 -- 172.21.15.97:0/1638013044 <== mon.1 v1:172.21.15.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2669613670 0 0) 0x7f72700b77a0 con 0x7f72700a98f0 2024-03-25T18:20:03.872 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:03.870+0000 7f7266ffd640 1 -- 172.21.15.97:0/1638013044 >> v1:172.21.15.97:6790/0 conn(0x7f72701430c0 legacy=0x7f72701dba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:03.872 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:03.870+0000 7f7266ffd640 1 -- 172.21.15.97:0/1638013044 >> v1:172.21.15.97:6789/0 conn(0x7f727013f710 legacy=0x7f72701db330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:03.872 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:03.870+0000 7f7266ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:20:03.872 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:03.870+0000 7f7266ffd640 1 monclient: found mon.b 2024-03-25T18:20:03.872 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:03.870+0000 7f7266ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:03.872 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:03.870+0000 7f7266ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:20:03.872 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:03.870+0000 7f7266ffd640 1 -- 172.21.15.97:0/1638013044 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72701dc150 con 0x7f72700a98f0 2024-03-25T18:20:03.872 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:03.870+0000 7f7266ffd640 10 monclient: _finish_auth 0 2024-03-25T18:20:03.872 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:03.870+0000 7f7266ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:20:03.873 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:03.870+0000 7f7266ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:03.873 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:03.870+0000 7f7274997640 5 monclient: authenticate success, global_id 4390 2024-03-25T18:20:03.873 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:03.870+0000 7f7274997640 10 monclient: _renew_subs 2024-03-25T18:20:03.873 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:03.870+0000 7f7274997640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:20:03.873 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:03.870+0000 7f7274997640 1 -- 172.21.15.97:0/1638013044 --> v1:172.21.15.98:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7270136ef0 con 0x7f72700a98f0 2024-03-25T18:20:03.873 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:03.870+0000 7f7274997640 10 client.4390.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:20:03.873 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:03.870+0000 7f7274997640 10 monclient: _renew_subs 2024-03-25T18:20:03.873 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:03.870+0000 7f7274997640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:20:03.873 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:03.870+0000 7f7274997640 1 -- 172.21.15.97:0/1638013044 --> v1:172.21.15.98:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7270137ca0 con 0x7f72700a98f0 2024-03-25T18:20:03.873 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:03.870+0000 7f7274997640 1 librados: init done 2024-03-25T18:20:03.874 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:03.871+0000 7f7274997640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:03.874 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:03.871+0000 7f7274997640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:03.874 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:03.871+0000 7f7274997640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:20:03.874 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:03.871+0000 7f7274997640 1 -- 172.21.15.97:0/1638013044 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72700627b0 con 0x7f72700a98f0 2024-03-25T18:20:04.145 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:04.144+0000 7f7266ffd640 1 -- 172.21.15.97:0/1638013044 <== mon.1 v1:172.21.15.98:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7260001c50 con 0x7f72700a98f0 2024-03-25T18:20:04.146 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:04.144+0000 7f7266ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:04.146 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:04.144+0000 7f7266ffd640 1 -- 172.21.15.97:0/1638013044 <== mon.1 v1:172.21.15.98:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f7260003a60 con 0x7f72700a98f0 2024-03-25T18:20:04.146 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:04.144+0000 7f7266ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:04.146 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:04.144+0000 7f7266ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:20:04.146 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:04.144+0000 7f7266ffd640 10 monclient: dump: 2024-03-25T18:20:04.146 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:20:04.146 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:20:04.146 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:20:04.146 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:20:04.146 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:04.146 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:20:04.146 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:20:04.146 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:20:04.146 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:20:04.147 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:20:04.147 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:04.144+0000 7f726d59a640 4 set_mon_vals no callback set 2024-03-25T18:20:04.147 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:04.145+0000 7f7266ffd640 1 -- 172.21.15.97:0/1638013044 <== mon.1 v1:172.21.15.98:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2377622137 0 0) 0x7f7260003c80 con 0x7f72700a98f0 2024-03-25T18:20:04.147 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:04.146+0000 7f7267fff640 10 monclient: get_auth_request con 0x7f725006ddb0 auth_method 0 2024-03-25T18:20:04.147 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:04.146+0000 7f7266ffd640 10 client.4390.objecter ms_handle_connect 0x7f725006ddb0 2024-03-25T18:20:04.152 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:04.150+0000 7f7266ffd640 1 -- 172.21.15.97:0/1638013044 <== mon.1 v1:172.21.15.98:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 430711918 0 0) 0x7f72600548f0 con 0x7f72700a98f0 2024-03-25T18:20:04.152 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:04.150+0000 7f7266ffd640 10 client.4390.objecter ms_dispatch 0x7f7270000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:20:04.152 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:04.150+0000 7f7266ffd640 3 client.4390.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:20:04.152 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:04.150+0000 7f7266ffd640 3 client.4390.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:20:04.152 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:04.150+0000 7f7266ffd640 20 client.4390.objecter dump_active .. 0 homeless 2024-03-25T18:20:04.152 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:04.150+0000 7f7266ffd640 1 -- 172.21.15.97:0/1638013044 <== mon.1 v1:172.21.15.98:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7260050400 con 0x7f72700a98f0 2024-03-25T18:20:04.152 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:04.151+0000 7f7266ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:04.152 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:04.151+0000 7f7266ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:20:04.409 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:04.408+0000 7f7274997640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:04.409 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:04.408+0000 7f7274997640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:04.409 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:04.408+0000 7f7274997640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:20:04.409 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:04.408+0000 7f7274997640 1 -- 172.21.15.97:0/1638013044 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f72700b7c80 con 0x7f72700a98f0 2024-03-25T18:20:04.566 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:04.564+0000 7f7266ffd640 1 -- 172.21.15.97:0/1638013044 <== mon.1 v1:172.21.15.98:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2169427412) 0x7f7260052330 con 0x7f72700a98f0 2024-03-25T18:20:04.566 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:04.564+0000 7f7266ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:04.566 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:04.564+0000 7f7266ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:20:04.566 INFO:teuthology.misc.health.smithi097.stdout: 2024-03-25T18:20:04.567 INFO:teuthology.misc.health.smithi097.stdout:{"epoch":10,"fsid":"4c3d472c-79b8-486b-9e3d-2738d4306754","created":"2024-03-25T18:16:14.737309+0000","modified":"2024-03-25T18:16:20.357492+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:20.357492+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cb5152b9-262f-45c3-94b8-18d0e81df558","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6daf8793-0838-473d-a7d6-9a0d2894c75d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"136f1493-3b9c-4f43-a2e7-641482d69dae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"397d538c-4f64-4daf-9e8b-662d09f91327","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4f1df08-663f-4bda-93f8-3878ab7c24b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c4b18d70-a3ce-4b15-9b70-eae5525bd761","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8dd4033a-de0e-4f8a-96dc-59f3c1f825b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0077722e-ffa6-4a3d-9891-929d9869b286","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:20:04.569 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:04.567+0000 7f7274997640 10 librados: watch_flush enter 2024-03-25T18:20:04.569 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:04.567+0000 7f7274997640 10 librados: watch_flush exit 2024-03-25T18:20:04.569 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:04.567+0000 7f7274997640 20 client.4390.objecter shutdown clearing up homeless session... 2024-03-25T18:20:04.569 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:04.567+0000 7f7274997640 10 client.4390.objecter successfully canceled tick 2024-03-25T18:20:04.569 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:04.567+0000 7f7274997640 1 -- 172.21.15.97:0/1638013044 >> v1:172.21.15.98:6800/800387573 conn(0x7f725006ddb0 legacy=0x7f7250070270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:04.569 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:04.567+0000 7f7274997640 10 monclient: shutdown 2024-03-25T18:20:04.569 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:04.567+0000 7f7274997640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:04.569 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:04.567+0000 7f7274997640 1 -- 172.21.15.97:0/1638013044 >> v1:172.21.15.98:6789/0 conn(0x7f72700a98f0 legacy=0x7f7270137590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:04.569 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:04.567+0000 7f7274997640 1 -- 172.21.15.97:0/1638013044 shutdown_connections 2024-03-25T18:20:04.569 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:04.567+0000 7f7274997640 1 -- 172.21.15.97:0/1638013044 >> 172.21.15.97:0/1638013044 conn(0x7f72700b7ea0 msgr2=0x7f7270074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:04.569 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:04.567+0000 7f7274997640 1 -- 172.21.15.97:0/1638013044 shutdown_connections 2024-03-25T18:20:04.569 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:04.567+0000 7f7274997640 1 -- 172.21.15.97:0/1638013044 wait complete. 2024-03-25T18:20:04.569 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:04.568+0000 7f7274997640 1 librados: shutdown 2024-03-25T18:20:04.581 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:20:11.387 DEBUG:teuthology.orchestra.run.smithi097:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:20:11.502 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.500+0000 7f4820f8f640 10 monclient: get_monmap_and_config 2024-03-25T18:20:11.502 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.500+0000 7f4820f8f640 10 monclient: build_initial_monmap 2024-03-25T18:20:11.502 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.500+0000 7f4820f8f640 10 monclient: monmap: 2024-03-25T18:20:11.502 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:20:11.502 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:20:11.502 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:20:11.502131+0000 2024-03-25T18:20:11.502 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:20:11.502131+0000 2024-03-25T18:20:11.503 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:11.503 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:20:11.503 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:20:11.503 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T18:20:11.503 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-b 2024-03-25T18:20:11.503 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:20:11.504 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.501+0000 7f4820f8f640 1 Processor -- start 2024-03-25T18:20:11.504 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.501+0000 7f4820f8f640 1 -- start start 2024-03-25T18:20:11.504 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.501+0000 7f4820f8f640 10 monclient: init 2024-03-25T18:20:11.504 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.502+0000 7f4820f8f640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:11.505 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.502+0000 7f4820f8f640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:20:11.505 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.502+0000 7f4820f8f640 10 monclient(hunting): picked mon.noname-a con 0x7f481c0b4f40 addr v1:172.21.15.97:6789/0 2024-03-25T18:20:11.505 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.502+0000 7f4820f8f640 10 monclient(hunting): picked mon.noname-b con 0x7f481c13fd90 addr v1:172.21.15.97:6790/0 2024-03-25T18:20:11.505 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.502+0000 7f4820f8f640 10 monclient(hunting): picked mon.noname-c con 0x7f481c0b4540 addr v1:172.21.15.98:6789/0 2024-03-25T18:20:11.505 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.502+0000 7f4820f8f640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f481c0aa0c0 con 0x7f481c0b4f40 2024-03-25T18:20:11.505 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.502+0000 7f4820f8f640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f481c0ab170 con 0x7f481c0b4540 2024-03-25T18:20:11.505 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.502+0000 7f4820f8f640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f481c0a9ba0 con 0x7f481c13fd90 2024-03-25T18:20:11.505 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.502+0000 7f4820f8f640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:11.505 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.502+0000 7f4820f8f640 10 monclient(hunting): authenticate will time out at 7823.778609s 2024-03-25T18:20:11.510 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.509+0000 7f4811d9b640 1 --1- >> v1:172.21.15.97:6790/0 conn(0x7f481c13fd90 0x7f481c1423f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6790/0 says I am v1:172.21.15.97:50490/0 (socket says 172.21.15.97:50490) 2024-03-25T18:20:11.510 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.509+0000 7f4811d9b640 1 -- 172.21.15.97:0/1321416499 learned_addr learned my addr 172.21.15.97:0/1321416499 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:20:11.521 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.520+0000 7f4819d9b640 1 -- 172.21.15.97:0/1321416499 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 595773584 0 0) 0x7f481c0ab170 con 0x7f481c0b4540 2024-03-25T18:20:11.521 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.520+0000 7f4819d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:20:11.521 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.520+0000 7f4819d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:11.522 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.520+0000 7f4819d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:11.522 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.520+0000 7f4819d9b640 10 monclient(hunting): my global_id is 4393 2024-03-25T18:20:11.522 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.520+0000 7f4819d9b640 1 -- 172.21.15.97:0/1321416499 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4808003520 con 0x7f481c0b4540 2024-03-25T18:20:11.529 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.528+0000 7f4819d9b640 1 -- 172.21.15.97:0/1321416499 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1994981628 0 0) 0x7f481c0a9ba0 con 0x7f481c13fd90 2024-03-25T18:20:11.529 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.528+0000 7f4819d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:20:11.529 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.528+0000 7f4819d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:11.529 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.528+0000 7f4819d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:11.530 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.528+0000 7f4819d9b640 10 monclient(hunting): my global_id is 4388 2024-03-25T18:20:11.530 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.528+0000 7f4819d9b640 1 -- 172.21.15.97:0/1321416499 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f481c0ab170 con 0x7f481c13fd90 2024-03-25T18:20:11.537 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.536+0000 7f4819d9b640 1 -- 172.21.15.97:0/1321416499 <== mon.2 v1:172.21.15.97:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 806622127 0 0) 0x7f481c0ab170 con 0x7f481c13fd90 2024-03-25T18:20:11.537 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.536+0000 7f4819d9b640 1 -- 172.21.15.97:0/1321416499 >> v1:172.21.15.98:6789/0 conn(0x7f481c0b4540 legacy=0x7f481c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:11.537 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.536+0000 7f4819d9b640 1 -- 172.21.15.97:0/1321416499 >> v1:172.21.15.97:6789/0 conn(0x7f481c0b4f40 legacy=0x7f481c13f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:11.538 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.536+0000 7f4819d9b640 10 monclient: _finish_hunting 0 2024-03-25T18:20:11.538 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.536+0000 7f4819d9b640 1 monclient: found mon.noname-b 2024-03-25T18:20:11.538 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.536+0000 7f4819d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:11.538 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.536+0000 7f4819d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.97:6790/0 2024-03-25T18:20:11.538 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.536+0000 7f4819d9b640 1 -- 172.21.15.97:0/1321416499 --> v1:172.21.15.97:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f481c0a9680 con 0x7f481c13fd90 2024-03-25T18:20:11.538 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.536+0000 7f4819d9b640 10 monclient: _finish_auth 0 2024-03-25T18:20:11.538 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.536+0000 7f4819d9b640 10 monclient: _check_auth_tickets 2024-03-25T18:20:11.538 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.536+0000 7f4819d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:11.538 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.536+0000 7f4820f8f640 5 monclient: authenticate success, global_id 4388 2024-03-25T18:20:11.538 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.536+0000 7f4820f8f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:20:11.539 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.537+0000 7f4819d9b640 1 -- 172.21.15.97:0/1321416499 <== mon.2 v1:172.21.15.97:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f4804001630 con 0x7f481c13fd90 2024-03-25T18:20:11.539 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.537+0000 7f4819d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:11.539 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.537+0000 7f4819d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:20:11.539 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.537+0000 7f4819d9b640 10 monclient: dump: 2024-03-25T18:20:11.539 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:20:11.539 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:20:11.539 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:20:11.539 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:20:11.539 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:11.540 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:20:11.540 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:20:11.540 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:20:11.540 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:20:11.540 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:20:11.540 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.537+0000 7f4820f8f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:20:11.549 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.548+0000 7f4819d9b640 1 -- 172.21.15.97:0/1321416499 <== mon.2 v1:172.21.15.97:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f48040018c0 con 0x7f481c13fd90 2024-03-25T18:20:11.549 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.548+0000 7f4819d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:11.550 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.548+0000 7f4819d9b640 1 -- 172.21.15.97:0/1321416499 <== mon.2 v1:172.21.15.97:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f4804001bd0 con 0x7f481c13fd90 2024-03-25T18:20:11.550 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.548+0000 7f4820f8f640 10 monclient: get_monmap_and_config success 2024-03-25T18:20:11.550 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.548+0000 7f4820f8f640 4 set_mon_vals no callback set 2024-03-25T18:20:11.550 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.548+0000 7f4819d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:11.550 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.548+0000 7f4819d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:20:11.550 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.548+0000 7f4819d9b640 10 monclient: dump: 2024-03-25T18:20:11.550 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:20:11.550 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:20:11.550 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:20:11.550 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:20:11.550 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:11.550 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:20:11.550 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:20:11.551 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:20:11.551 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:20:11.551 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:20:11.552 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.549+0000 7f4820f8f640 10 monclient: shutdown 2024-03-25T18:20:11.552 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.549+0000 7f4820f8f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:11.552 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.549+0000 7f4820f8f640 1 -- 172.21.15.97:0/1321416499 >> v1:172.21.15.97:6790/0 conn(0x7f481c13fd90 legacy=0x7f481c1423f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:11.552 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.549+0000 7f4820f8f640 1 -- 172.21.15.97:0/1321416499 shutdown_connections 2024-03-25T18:20:11.552 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.549+0000 7f4820f8f640 1 -- 172.21.15.97:0/1321416499 >> 172.21.15.97:0/1321416499 conn(0x7f481c0617a0 msgr2=0x7f481c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:11.552 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.549+0000 7f4820f8f640 1 -- 172.21.15.97:0/1321416499 shutdown_connections 2024-03-25T18:20:11.552 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.549+0000 7f4820f8f640 1 -- 172.21.15.97:0/1321416499 wait complete. 2024-03-25T18:20:11.552 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.550+0000 7f4820f8f640 10 monclient: build_initial_monmap 2024-03-25T18:20:11.552 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.550+0000 7f4820f8f640 10 monclient: monmap: 2024-03-25T18:20:11.552 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:20:11.552 INFO:teuthology.misc.health.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:20:11.553 INFO:teuthology.misc.health.smithi097.stderr:last_changed 0.000000 2024-03-25T18:20:11.553 INFO:teuthology.misc.health.smithi097.stderr:created 0.000000 2024-03-25T18:20:11.553 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:11.553 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 1 2024-03-25T18:20:11.553 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:20:11.553 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T18:20:11.553 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-c 2024-03-25T18:20:11.553 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:20:11.553 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.550+0000 7f4820f8f640 1 librados: starting msgr at 2024-03-25T18:20:11.553 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.550+0000 7f4820f8f640 1 librados: starting objecter 2024-03-25T18:20:11.553 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.551+0000 7f4820f8f640 1 Processor -- start 2024-03-25T18:20:11.553 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.551+0000 7f4820f8f640 1 -- start start 2024-03-25T18:20:11.553 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.551+0000 7f4820f8f640 1 librados: setting wanted keys 2024-03-25T18:20:11.553 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.551+0000 7f4820f8f640 1 librados: calling monclient init 2024-03-25T18:20:11.553 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.551+0000 7f4820f8f640 10 monclient: init 2024-03-25T18:20:11.554 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.551+0000 7f4820f8f640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:11.554 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.551+0000 7f4820f8f640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:20:11.554 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.551+0000 7f4820f8f640 10 monclient(hunting): picked mon.noname-b con 0x7f481c0b4540 addr v1:172.21.15.98:6789/0 2024-03-25T18:20:11.555 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.551+0000 7f4820f8f640 10 monclient(hunting): picked mon.noname-a con 0x7f481c0b4f40 addr v1:172.21.15.97:6789/0 2024-03-25T18:20:11.555 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.551+0000 7f4820f8f640 10 monclient(hunting): picked mon.noname-c con 0x7f481c13fd90 addr v1:172.21.15.97:6790/0 2024-03-25T18:20:11.555 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.551+0000 7f4820f8f640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f481c0aebd0 con 0x7f481c0b4f40 2024-03-25T18:20:11.555 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.551+0000 7f4820f8f640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f481c0b0c00 con 0x7f481c0b4540 2024-03-25T18:20:11.555 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.551+0000 7f4820f8f640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f481c1380f0 con 0x7f481c13fd90 2024-03-25T18:20:11.555 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.551+0000 7f4820f8f640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:11.555 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.551+0000 7f4820f8f640 10 monclient(hunting): authenticate will time out at 7823.827767s 2024-03-25T18:20:11.560 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.559+0000 7f481a59c640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f481c0b4540 0x7f481c0af340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.97:44582/0 (socket says 172.21.15.97:44582) 2024-03-25T18:20:11.560 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.559+0000 7f481a59c640 1 -- 172.21.15.97:0/2082022244 learned_addr learned my addr 172.21.15.97:0/2082022244 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:20:11.562 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.561+0000 7f48137fe640 10 client.?.objecter ms_handle_connect 0x7f481c0b4540 2024-03-25T18:20:11.562 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.561+0000 7f48137fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:11.563 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.562+0000 7f48137fe640 10 client.?.objecter ms_handle_connect 0x7f481c13fd90 2024-03-25T18:20:11.563 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.562+0000 7f48137fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:11.570 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.569+0000 7f48137fe640 1 -- 172.21.15.97:0/2082022244 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1287439774 0 0) 0x7f481c1380f0 con 0x7f481c13fd90 2024-03-25T18:20:11.570 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.569+0000 7f48137fe640 10 monclient(hunting): _negotiate 2024-03-25T18:20:11.570 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.569+0000 7f48137fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:11.570 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.569+0000 7f48137fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:11.570 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.569+0000 7f48137fe640 10 monclient(hunting): my global_id is 4391 2024-03-25T18:20:11.570 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.569+0000 7f48137fe640 1 -- 172.21.15.97:0/2082022244 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47fc004eb0 con 0x7f481c13fd90 2024-03-25T18:20:11.574 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.572+0000 7f48137fe640 1 -- 172.21.15.97:0/2082022244 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1579866297 0 0) 0x7f481c0b0c00 con 0x7f481c0b4540 2024-03-25T18:20:11.574 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.572+0000 7f48137fe640 10 monclient(hunting): _negotiate 2024-03-25T18:20:11.574 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.572+0000 7f48137fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:11.574 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.572+0000 7f48137fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:11.574 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.572+0000 7f48137fe640 10 monclient(hunting): my global_id is 4396 2024-03-25T18:20:11.574 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.572+0000 7f48137fe640 1 -- 172.21.15.97:0/2082022244 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f481c1380f0 con 0x7f481c0b4540 2024-03-25T18:20:11.580 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.579+0000 7f48137fe640 10 client.?.objecter ms_handle_connect 0x7f481c0b4f40 2024-03-25T18:20:11.580 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.579+0000 7f48137fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:11.596 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.594+0000 7f48137fe640 1 -- 172.21.15.97:0/2082022244 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2447933862 0 0) 0x7f481c1380f0 con 0x7f481c0b4540 2024-03-25T18:20:11.596 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.594+0000 7f48137fe640 1 -- 172.21.15.97:0/2082022244 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f481c0b0c00 con 0x7f481c0b4540 2024-03-25T18:20:11.596 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.594+0000 7f48137fe640 1 -- 172.21.15.97:0/2082022244 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f480c001990 con 0x7f481c0b4540 2024-03-25T18:20:11.596 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.594+0000 7f48137fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:11.596 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.594+0000 7f48137fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:20:11.596 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.594+0000 7f48137fe640 10 monclient(hunting): dump: 2024-03-25T18:20:11.596 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:20:11.596 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:20:11.596 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:20:11.596 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:20:11.596 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:11.596 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:20:11.596 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:20:11.597 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:20:11.597 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:20:11.597 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:20:11.597 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.594+0000 7f48137fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:20:11.598 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.596+0000 7f48137fe640 1 -- 172.21.15.97:0/2082022244 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3678239507 0 0) 0x7f481c0aebd0 con 0x7f481c0b4f40 2024-03-25T18:20:11.598 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.597+0000 7f48137fe640 10 monclient(hunting): _negotiate 2024-03-25T18:20:11.598 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.597+0000 7f48137fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:11.598 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.597+0000 7f48137fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:11.598 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.597+0000 7f48137fe640 10 monclient(hunting): my global_id is 4398 2024-03-25T18:20:11.598 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.597+0000 7f48137fe640 1 -- 172.21.15.97:0/2082022244 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f481c1380f0 con 0x7f481c0b4f40 2024-03-25T18:20:11.604 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.602+0000 7f48137fe640 1 -- 172.21.15.97:0/2082022244 <== mon.2 v1:172.21.15.97:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1473794134 0 0) 0x7f47fc004eb0 con 0x7f481c13fd90 2024-03-25T18:20:11.604 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.602+0000 7f48137fe640 1 -- 172.21.15.97:0/2082022244 --> v1:172.21.15.97:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f481c0aebd0 con 0x7f481c13fd90 2024-03-25T18:20:11.604 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.602+0000 7f48137fe640 1 -- 172.21.15.97:0/2082022244 <== mon.2 v1:172.21.15.97:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f4814001cd0 con 0x7f481c13fd90 2024-03-25T18:20:11.604 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.602+0000 7f48137fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:11.604 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.602+0000 7f48137fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:20:11.604 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.602+0000 7f48137fe640 10 monclient(hunting): dump: 2024-03-25T18:20:11.604 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:20:11.604 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:20:11.604 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:20:11.604 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:20:11.604 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:11.604 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:20:11.605 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:20:11.605 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:20:11.605 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:20:11.605 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:20:11.606 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.604+0000 7f48137fe640 1 -- 172.21.15.97:0/2082022244 <== mon.1 v1:172.21.15.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3248176625 0 0) 0x7f481c0b0c00 con 0x7f481c0b4540 2024-03-25T18:20:11.606 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.604+0000 7f48137fe640 1 -- 172.21.15.97:0/2082022244 >> v1:172.21.15.97:6790/0 conn(0x7f481c13fd90 legacy=0x7f481c1379e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:11.606 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.604+0000 7f48137fe640 1 -- 172.21.15.97:0/2082022244 >> v1:172.21.15.97:6789/0 conn(0x7f481c0b4f40 legacy=0x7f481c1372d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:11.606 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.604+0000 7f48137fe640 10 monclient: _finish_hunting 0 2024-03-25T18:20:11.606 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.604+0000 7f48137fe640 1 monclient: found mon.b 2024-03-25T18:20:11.607 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.604+0000 7f48137fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:11.607 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.604+0000 7f48137fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:20:11.607 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.604+0000 7f48137fe640 1 -- 172.21.15.97:0/2082022244 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f481c0b0e70 con 0x7f481c0b4540 2024-03-25T18:20:11.607 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.604+0000 7f48137fe640 10 monclient: _finish_auth 0 2024-03-25T18:20:11.607 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.604+0000 7f48137fe640 10 monclient: _check_auth_tickets 2024-03-25T18:20:11.607 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.604+0000 7f48137fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:11.607 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.604+0000 7f4820f8f640 5 monclient: authenticate success, global_id 4396 2024-03-25T18:20:11.607 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.604+0000 7f4820f8f640 10 monclient: _renew_subs 2024-03-25T18:20:11.607 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.604+0000 7f4820f8f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:20:11.607 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.604+0000 7f4820f8f640 1 -- 172.21.15.97:0/2082022244 --> v1:172.21.15.98:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f481c0ae810 con 0x7f481c0b4540 2024-03-25T18:20:11.607 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.604+0000 7f4820f8f640 10 client.4396.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:20:11.607 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.604+0000 7f4820f8f640 10 monclient: _renew_subs 2024-03-25T18:20:11.607 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.604+0000 7f4820f8f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:20:11.607 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.604+0000 7f4820f8f640 1 -- 172.21.15.97:0/2082022244 --> v1:172.21.15.98:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f481c0afa50 con 0x7f481c0b4540 2024-03-25T18:20:11.607 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.604+0000 7f4820f8f640 1 librados: init done 2024-03-25T18:20:11.608 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.605+0000 7f4820f8f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:11.608 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.605+0000 7f4820f8f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:11.608 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.605+0000 7f4820f8f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:20:11.608 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.605+0000 7f4820f8f640 1 -- 172.21.15.97:0/2082022244 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f481c0afd50 con 0x7f481c0b4540 2024-03-25T18:20:11.882 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.881+0000 7f48137fe640 1 -- 172.21.15.97:0/2082022244 <== mon.1 v1:172.21.15.98:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f480c0035f0 con 0x7f481c0b4540 2024-03-25T18:20:11.883 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.881+0000 7f48137fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:11.883 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.881+0000 7f48137fe640 1 -- 172.21.15.97:0/2082022244 <== mon.1 v1:172.21.15.98:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f480c0039c0 con 0x7f481c0b4540 2024-03-25T18:20:11.883 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.881+0000 7f48137fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:11.883 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.881+0000 7f48137fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:20:11.883 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.881+0000 7f48137fe640 10 monclient: dump: 2024-03-25T18:20:11.883 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:20:11.883 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:20:11.883 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:20:11.883 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:20:11.883 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:11.883 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:20:11.884 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:20:11.884 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:20:11.884 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:20:11.884 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:20:11.884 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.881+0000 7f481959a640 4 set_mon_vals no callback set 2024-03-25T18:20:11.885 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.882+0000 7f48137fe640 1 -- 172.21.15.97:0/2082022244 <== mon.1 v1:172.21.15.98:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2377622137 0 0) 0x7f480c003be0 con 0x7f481c0b4540 2024-03-25T18:20:11.885 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.882+0000 7f4811d9b640 10 monclient: get_auth_request con 0x7f47fc06dfb0 auth_method 0 2024-03-25T18:20:11.885 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.883+0000 7f48137fe640 10 client.4396.objecter ms_handle_connect 0x7f47fc06dfb0 2024-03-25T18:20:11.889 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.887+0000 7f48137fe640 1 -- 172.21.15.97:0/2082022244 <== mon.1 v1:172.21.15.98:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 430711918 0 0) 0x7f480c054770 con 0x7f481c0b4540 2024-03-25T18:20:11.889 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.887+0000 7f48137fe640 10 client.4396.objecter ms_dispatch 0x7f481c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:20:11.889 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.887+0000 7f48137fe640 3 client.4396.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:20:11.889 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.887+0000 7f48137fe640 3 client.4396.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:20:11.889 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.887+0000 7f48137fe640 20 client.4396.objecter dump_active .. 0 homeless 2024-03-25T18:20:11.892 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.891+0000 7f48137fe640 1 -- 172.21.15.97:0/2082022244 <== mon.1 v1:172.21.15.98:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f480c050280 con 0x7f481c0b4540 2024-03-25T18:20:11.892 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.891+0000 7f48137fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:11.892 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:11.891+0000 7f48137fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:20:12.151 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:12.150+0000 7f4820f8f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:12.151 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:12.150+0000 7f4820f8f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:12.151 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:12.150+0000 7f4820f8f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:20:12.151 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:12.150+0000 7f4820f8f640 1 -- 172.21.15.97:0/2082022244 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f481c0627b0 con 0x7f481c0b4540 2024-03-25T18:20:12.181 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:12.180+0000 7f48137fe640 1 -- 172.21.15.97:0/2082022244 <== mon.1 v1:172.21.15.98:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2169427412) 0x7f480c0521b0 con 0x7f481c0b4540 2024-03-25T18:20:12.181 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:12.180+0000 7f48137fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:12.182 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:12.180+0000 7f48137fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:20:12.182 INFO:teuthology.misc.health.smithi097.stdout: 2024-03-25T18:20:12.182 INFO:teuthology.misc.health.smithi097.stdout:{"epoch":10,"fsid":"4c3d472c-79b8-486b-9e3d-2738d4306754","created":"2024-03-25T18:16:14.737309+0000","modified":"2024-03-25T18:16:20.357492+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:20.357492+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cb5152b9-262f-45c3-94b8-18d0e81df558","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6daf8793-0838-473d-a7d6-9a0d2894c75d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"136f1493-3b9c-4f43-a2e7-641482d69dae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"397d538c-4f64-4daf-9e8b-662d09f91327","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4f1df08-663f-4bda-93f8-3878ab7c24b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c4b18d70-a3ce-4b15-9b70-eae5525bd761","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8dd4033a-de0e-4f8a-96dc-59f3c1f825b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0077722e-ffa6-4a3d-9891-929d9869b286","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:20:12.184 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:12.182+0000 7f4820f8f640 10 librados: watch_flush enter 2024-03-25T18:20:12.184 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:12.182+0000 7f4820f8f640 10 librados: watch_flush exit 2024-03-25T18:20:12.185 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:12.182+0000 7f4820f8f640 20 client.4396.objecter shutdown clearing up homeless session... 2024-03-25T18:20:12.185 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:12.182+0000 7f4820f8f640 10 client.4396.objecter successfully canceled tick 2024-03-25T18:20:12.185 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:12.182+0000 7f4820f8f640 1 -- 172.21.15.97:0/2082022244 >> v1:172.21.15.98:6800/800387573 conn(0x7f47fc06dfb0 legacy=0x7f47fc070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:12.185 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:12.182+0000 7f4820f8f640 10 monclient: shutdown 2024-03-25T18:20:12.185 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:12.182+0000 7f4820f8f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:12.185 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:12.182+0000 7f4820f8f640 1 -- 172.21.15.97:0/2082022244 >> v1:172.21.15.98:6789/0 conn(0x7f481c0b4540 legacy=0x7f481c0af340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:12.185 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:12.182+0000 7f4820f8f640 1 -- 172.21.15.97:0/2082022244 shutdown_connections 2024-03-25T18:20:12.185 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:12.183+0000 7f4820f8f640 1 -- 172.21.15.97:0/2082022244 >> 172.21.15.97:0/2082022244 conn(0x7f481c0617a0 msgr2=0x7f481c0b3130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:12.185 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:12.183+0000 7f4820f8f640 1 -- 172.21.15.97:0/2082022244 shutdown_connections 2024-03-25T18:20:12.186 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:12.183+0000 7f4820f8f640 1 -- 172.21.15.97:0/2082022244 wait complete. 2024-03-25T18:20:12.186 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:12.183+0000 7f4820f8f640 1 librados: shutdown 2024-03-25T18:20:12.197 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:20:19.043 DEBUG:teuthology.orchestra.run.smithi097:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:20:19.162 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.160+0000 7f1fe604a640 10 monclient: get_monmap_and_config 2024-03-25T18:20:19.162 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.160+0000 7f1fe604a640 10 monclient: build_initial_monmap 2024-03-25T18:20:19.162 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.160+0000 7f1fe604a640 10 monclient: monmap: 2024-03-25T18:20:19.162 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:20:19.162 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:20:19.162 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:20:19.162147+0000 2024-03-25T18:20:19.162 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:20:19.162147+0000 2024-03-25T18:20:19.162 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:19.162 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:20:19.162 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:20:19.162 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T18:20:19.163 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-b 2024-03-25T18:20:19.163 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:20:19.164 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.161+0000 7f1fe604a640 1 Processor -- start 2024-03-25T18:20:19.164 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.161+0000 7f1fe604a640 1 -- start start 2024-03-25T18:20:19.164 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.161+0000 7f1fe604a640 10 monclient: init 2024-03-25T18:20:19.165 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.162+0000 7f1fe604a640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:19.165 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.162+0000 7f1fe604a640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:20:19.165 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.162+0000 7f1fe604a640 10 monclient(hunting): picked mon.noname-b con 0x7f1fe0061830 addr v1:172.21.15.97:6790/0 2024-03-25T18:20:19.165 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.162+0000 7f1fe604a640 10 monclient(hunting): picked mon.noname-c con 0x7f1fe013c9f0 addr v1:172.21.15.98:6789/0 2024-03-25T18:20:19.165 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.162+0000 7f1fe604a640 10 monclient(hunting): picked mon.noname-a con 0x7f1fe0072af0 addr v1:172.21.15.97:6789/0 2024-03-25T18:20:19.165 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.162+0000 7f1fe604a640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fe0071fc0 con 0x7f1fe0072af0 2024-03-25T18:20:19.165 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.163+0000 7f1fe604a640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fe00707a0 con 0x7f1fe013c9f0 2024-03-25T18:20:19.165 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.163+0000 7f1fe604a640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fe00ae9e0 con 0x7f1fe0061830 2024-03-25T18:20:19.165 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.163+0000 7f1fe604a640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:19.165 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.163+0000 7f1fe604a640 10 monclient(hunting): authenticate will time out at 7831.438933s 2024-03-25T18:20:19.171 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.170+0000 7f1fdf7fe640 1 --1- >> v1:172.21.15.97:6790/0 conn(0x7f1fe0061830 0x7f1fe013c2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6790/0 says I am v1:172.21.15.97:55324/0 (socket says 172.21.15.97:55324) 2024-03-25T18:20:19.171 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.170+0000 7f1fdf7fe640 1 -- 172.21.15.97:0/1795332585 learned_addr learned my addr 172.21.15.97:0/1795332585 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:20:19.172 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.170+0000 7f1fdf7fe640 1 -- 172.21.15.97:0/1795332585 >> v1:172.21.15.97:6790/0 conn(0x7f1fe0061830 legacy=0x7f1fe013c2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-25T18:20:19.172 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.170+0000 7f1fdf7fe640 1 -- 172.21.15.97:0/1795332585 >> v1:172.21.15.97:6790/0 conn(0x7f1fe0061830 legacy=0x7f1fe013c2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-25T18:20:19.172 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.170+0000 7f1fdf7fe640 1 --1- 172.21.15.97:0/1795332585 >> v1:172.21.15.97:6790/0 conn(0x7f1fe0061830 0x7f1fe013c2e0 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=0).handle_connect_reply_1 read connect reply failed 2024-03-25T18:20:19.190 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.189+0000 7f1fde7fc640 1 -- 172.21.15.97:0/1795332585 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 137999430 0 0) 0x7f1fe0071fc0 con 0x7f1fe0072af0 2024-03-25T18:20:19.191 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.189+0000 7f1fde7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:20:19.191 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.189+0000 7f1fde7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:19.191 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.189+0000 7f1fde7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:19.191 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.189+0000 7f1fde7fc640 10 monclient(hunting): my global_id is 4401 2024-03-25T18:20:19.191 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.189+0000 7f1fde7fc640 1 -- 172.21.15.97:0/1795332585 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1fc8005730 con 0x7f1fe0072af0 2024-03-25T18:20:19.206 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.204+0000 7f1fde7fc640 1 -- 172.21.15.97:0/1795332585 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3337554013 0 0) 0x7f1fe00707a0 con 0x7f1fe013c9f0 2024-03-25T18:20:19.206 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.204+0000 7f1fde7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:20:19.206 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.204+0000 7f1fde7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:19.206 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.204+0000 7f1fde7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:19.206 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.204+0000 7f1fde7fc640 10 monclient(hunting): my global_id is 4399 2024-03-25T18:20:19.206 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.204+0000 7f1fde7fc640 1 -- 172.21.15.97:0/1795332585 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1fe0071fc0 con 0x7f1fe013c9f0 2024-03-25T18:20:19.207 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.205+0000 7f1fde7fc640 1 -- 172.21.15.97:0/1795332585 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1442787948 0 0) 0x7f1fc8005730 con 0x7f1fe0072af0 2024-03-25T18:20:19.207 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.205+0000 7f1fde7fc640 1 -- 172.21.15.97:0/1795332585 >> v1:172.21.15.97:6790/0 conn(0x7f1fe0061830 legacy=0x7f1fe013c2e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:20:19.207 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.205+0000 7f1fde7fc640 1 -- 172.21.15.97:0/1795332585 >> v1:172.21.15.98:6789/0 conn(0x7f1fe013c9f0 legacy=0x7f1fe00723e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:19.207 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.205+0000 7f1fde7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:20:19.207 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.205+0000 7f1fde7fc640 1 monclient: found mon.noname-a 2024-03-25T18:20:19.207 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.205+0000 7f1fde7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:19.207 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.205+0000 7f1fde7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.97:6789/0 2024-03-25T18:20:19.207 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.205+0000 7f1fde7fc640 1 -- 172.21.15.97:0/1795332585 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fe0070cc0 con 0x7f1fe0072af0 2024-03-25T18:20:19.207 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.205+0000 7f1fde7fc640 10 monclient: _finish_auth 0 2024-03-25T18:20:19.207 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.205+0000 7f1fde7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:20:19.207 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.205+0000 7f1fde7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:19.207 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.205+0000 7f1fde7fc640 1 -- 172.21.15.97:0/1795332585 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f1fd8001a10 con 0x7f1fe0072af0 2024-03-25T18:20:19.207 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.205+0000 7f1fde7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:19.208 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.205+0000 7f1fde7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:20:19.208 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.205+0000 7f1fde7fc640 10 monclient: dump: 2024-03-25T18:20:19.208 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:20:19.208 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:20:19.208 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:20:19.208 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:20:19.208 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:19.208 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:20:19.208 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:20:19.208 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:20:19.208 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:20:19.208 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:20:19.208 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.205+0000 7f1fe604a640 5 monclient: authenticate success, global_id 4401 2024-03-25T18:20:19.208 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.205+0000 7f1fe604a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:20:19.217 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.215+0000 7f1fde7fc640 1 -- 172.21.15.97:0/1795332585 <== mon.0 v1:172.21.15.97:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1fd8001b70 con 0x7f1fe0072af0 2024-03-25T18:20:19.217 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.215+0000 7f1fde7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:19.217 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.215+0000 7f1fde7fc640 1 -- 172.21.15.97:0/1795332585 <== mon.0 v1:172.21.15.97:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f1fd8001f80 con 0x7f1fe0072af0 2024-03-25T18:20:19.217 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.215+0000 7f1fde7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:19.217 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.215+0000 7f1fde7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:20:19.217 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.215+0000 7f1fde7fc640 10 monclient: dump: 2024-03-25T18:20:19.217 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:20:19.217 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:20:19.217 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:20:19.217 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:20:19.217 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:19.217 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:20:19.217 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:20:19.218 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:20:19.218 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:20:19.218 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:20:19.218 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.215+0000 7f1fe604a640 10 monclient: get_monmap_and_config success 2024-03-25T18:20:19.218 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.215+0000 7f1fe604a640 4 set_mon_vals no callback set 2024-03-25T18:20:19.220 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.216+0000 7f1fe604a640 10 monclient: shutdown 2024-03-25T18:20:19.220 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.216+0000 7f1fe604a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:19.220 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.216+0000 7f1fe604a640 1 -- 172.21.15.97:0/1795332585 >> v1:172.21.15.97:6789/0 conn(0x7f1fe0072af0 legacy=0x7f1fe014a890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:19.220 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.216+0000 7f1fe604a640 1 -- 172.21.15.97:0/1795332585 shutdown_connections 2024-03-25T18:20:19.220 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.216+0000 7f1fe604a640 1 -- 172.21.15.97:0/1795332585 >> 172.21.15.97:0/1795332585 conn(0x7f1fe00b0ef0 msgr2=0x7f1fe00b12d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:19.220 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.216+0000 7f1fe604a640 1 -- 172.21.15.97:0/1795332585 shutdown_connections 2024-03-25T18:20:19.220 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.216+0000 7f1fe604a640 1 -- 172.21.15.97:0/1795332585 wait complete. 2024-03-25T18:20:19.220 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.217+0000 7f1fe604a640 10 monclient: build_initial_monmap 2024-03-25T18:20:19.220 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.217+0000 7f1fe604a640 10 monclient: monmap: 2024-03-25T18:20:19.220 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:20:19.221 INFO:teuthology.misc.health.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:20:19.221 INFO:teuthology.misc.health.smithi097.stderr:last_changed 0.000000 2024-03-25T18:20:19.221 INFO:teuthology.misc.health.smithi097.stderr:created 0.000000 2024-03-25T18:20:19.221 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:19.221 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 1 2024-03-25T18:20:19.221 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:20:19.221 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T18:20:19.221 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-c 2024-03-25T18:20:19.221 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:20:19.221 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.217+0000 7f1fe604a640 1 librados: starting msgr at 2024-03-25T18:20:19.221 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.217+0000 7f1fe604a640 1 librados: starting objecter 2024-03-25T18:20:19.221 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.217+0000 7f1fe604a640 1 Processor -- start 2024-03-25T18:20:19.221 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.217+0000 7f1fe604a640 1 -- start start 2024-03-25T18:20:19.221 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.217+0000 7f1fe604a640 1 librados: setting wanted keys 2024-03-25T18:20:19.222 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.217+0000 7f1fe604a640 1 librados: calling monclient init 2024-03-25T18:20:19.222 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.217+0000 7f1fe604a640 10 monclient: init 2024-03-25T18:20:19.222 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.218+0000 7f1fe604a640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:19.222 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.218+0000 7f1fe604a640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:20:19.222 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.218+0000 7f1fe604a640 10 monclient(hunting): picked mon.noname-c con 0x7f1fe0061830 addr v1:172.21.15.97:6790/0 2024-03-25T18:20:19.222 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.218+0000 7f1fe604a640 10 monclient(hunting): picked mon.noname-a con 0x7f1fe0072af0 addr v1:172.21.15.97:6789/0 2024-03-25T18:20:19.222 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.218+0000 7f1fe604a640 10 monclient(hunting): picked mon.noname-b con 0x7f1fe013c9f0 addr v1:172.21.15.98:6789/0 2024-03-25T18:20:19.222 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.218+0000 7f1fe604a640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fe0070210 con 0x7f1fe0072af0 2024-03-25T18:20:19.222 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.218+0000 7f1fe604a640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fe013de80 con 0x7f1fe013c9f0 2024-03-25T18:20:19.222 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.218+0000 7f1fe604a640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fe00b29e0 con 0x7f1fe0061830 2024-03-25T18:20:19.222 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.218+0000 7f1fe604a640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:19.222 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.218+0000 7f1fe604a640 10 monclient(hunting): authenticate will time out at 7831.494142s 2024-03-25T18:20:19.229 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.228+0000 7f1fdeffd640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f1fe0072af0 0x7f1fe0133110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:35382/0 (socket says 172.21.15.97:35382) 2024-03-25T18:20:19.229 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.228+0000 7f1fdeffd640 1 -- 172.21.15.97:0/2151965549 learned_addr learned my addr 172.21.15.97:0/2151965549 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:20:19.233 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.232+0000 7f1fc7fff640 10 client.?.objecter ms_handle_connect 0x7f1fe0072af0 2024-03-25T18:20:19.233 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.232+0000 7f1fc7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:19.235 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.234+0000 7f1fc7fff640 10 client.?.objecter ms_handle_connect 0x7f1fe013c9f0 2024-03-25T18:20:19.235 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.234+0000 7f1fc7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:19.236 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.234+0000 7f1fc7fff640 10 client.?.objecter ms_handle_connect 0x7f1fe0061830 2024-03-25T18:20:19.236 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.234+0000 7f1fc7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:19.242 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.240+0000 7f1fc7fff640 1 -- 172.21.15.97:0/2151965549 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1115802695 0 0) 0x7f1fe0070210 con 0x7f1fe0072af0 2024-03-25T18:20:19.242 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.240+0000 7f1fc7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:20:19.242 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.240+0000 7f1fc7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:19.242 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.240+0000 7f1fc7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:19.242 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.240+0000 7f1fc7fff640 10 monclient(hunting): my global_id is 4404 2024-03-25T18:20:19.242 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.240+0000 7f1fc7fff640 1 -- 172.21.15.97:0/2151965549 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1fcc004eb0 con 0x7f1fe0072af0 2024-03-25T18:20:19.242 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.240+0000 7f1fc7fff640 1 -- 172.21.15.97:0/2151965549 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1595534401 0 0) 0x7f1fe00b29e0 con 0x7f1fe0061830 2024-03-25T18:20:19.242 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.240+0000 7f1fc7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:20:19.242 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.240+0000 7f1fc7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:19.242 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.240+0000 7f1fc7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:19.242 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.240+0000 7f1fc7fff640 10 monclient(hunting): my global_id is 4394 2024-03-25T18:20:19.243 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.240+0000 7f1fc7fff640 1 -- 172.21.15.97:0/2151965549 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1fe0070210 con 0x7f1fe0061830 2024-03-25T18:20:19.248 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.246+0000 7f1fc7fff640 1 -- 172.21.15.97:0/2151965549 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4045764047 0 0) 0x7f1fe013de80 con 0x7f1fe013c9f0 2024-03-25T18:20:19.248 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.247+0000 7f1fc7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:20:19.248 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.247+0000 7f1fc7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:19.248 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.247+0000 7f1fc7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:19.248 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.247+0000 7f1fc7fff640 10 monclient(hunting): my global_id is 4402 2024-03-25T18:20:19.248 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.247+0000 7f1fc7fff640 1 -- 172.21.15.97:0/2151965549 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1fe00b29e0 con 0x7f1fe013c9f0 2024-03-25T18:20:19.252 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.251+0000 7f1fc7fff640 1 -- 172.21.15.97:0/2151965549 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 564305294 0 0) 0x7f1fcc004eb0 con 0x7f1fe0072af0 2024-03-25T18:20:19.252 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.251+0000 7f1fc7fff640 1 -- 172.21.15.97:0/2151965549 --> v1:172.21.15.97:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1fe013de80 con 0x7f1fe0072af0 2024-03-25T18:20:19.252 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.251+0000 7f1fc7fff640 1 -- 172.21.15.97:0/2151965549 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f1fd4001a10 con 0x7f1fe0072af0 2024-03-25T18:20:19.252 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.251+0000 7f1fc7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:19.252 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.251+0000 7f1fc7fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:20:19.253 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.251+0000 7f1fc7fff640 10 monclient(hunting): dump: 2024-03-25T18:20:19.253 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:20:19.253 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:20:19.253 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:20:19.253 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:20:19.253 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:19.253 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:20:19.253 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:20:19.253 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:20:19.253 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:20:19.253 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:20:19.253 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.251+0000 7f1fc7fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:20:19.260 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.258+0000 7f1fc7fff640 1 -- 172.21.15.97:0/2151965549 <== mon.2 v1:172.21.15.97:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2913435381 0 0) 0x7f1fe0070210 con 0x7f1fe0061830 2024-03-25T18:20:19.260 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.258+0000 7f1fc7fff640 1 -- 172.21.15.97:0/2151965549 --> v1:172.21.15.97:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1fcc004eb0 con 0x7f1fe0061830 2024-03-25T18:20:19.260 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.258+0000 7f1fc7fff640 1 -- 172.21.15.97:0/2151965549 <== mon.2 v1:172.21.15.97:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f1fd00014d0 con 0x7f1fe0061830 2024-03-25T18:20:19.260 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.258+0000 7f1fc7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:19.261 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.258+0000 7f1fc7fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:20:19.261 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.258+0000 7f1fc7fff640 10 monclient(hunting): dump: 2024-03-25T18:20:19.261 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:20:19.261 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:20:19.261 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:20:19.261 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:20:19.261 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:19.261 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:20:19.261 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:20:19.261 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:20:19.261 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:20:19.261 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:20:19.261 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.259+0000 7f1fc7fff640 1 -- 172.21.15.97:0/2151965549 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3909962961 0 0) 0x7f1fe00b29e0 con 0x7f1fe013c9f0 2024-03-25T18:20:19.262 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.259+0000 7f1fc7fff640 1 -- 172.21.15.97:0/2151965549 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1fe0070210 con 0x7f1fe013c9f0 2024-03-25T18:20:19.262 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.259+0000 7f1fc7fff640 1 -- 172.21.15.97:0/2151965549 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f1fd8001e80 con 0x7f1fe013c9f0 2024-03-25T18:20:19.262 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.259+0000 7f1fc7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:19.262 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.259+0000 7f1fc7fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:20:19.262 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.259+0000 7f1fc7fff640 10 monclient(hunting): dump: 2024-03-25T18:20:19.262 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:20:19.262 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:20:19.262 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:20:19.262 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:20:19.262 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:19.262 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:20:19.262 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:20:19.262 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:20:19.262 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:20:19.263 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:20:19.266 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.264+0000 7f1fc7fff640 1 -- 172.21.15.97:0/2151965549 <== mon.0 v1:172.21.15.97:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2854385561 0 0) 0x7f1fe013de80 con 0x7f1fe0072af0 2024-03-25T18:20:19.266 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.264+0000 7f1fc7fff640 1 -- 172.21.15.97:0/2151965549 >> v1:172.21.15.97:6790/0 conn(0x7f1fe0061830 legacy=0x7f1fe0132490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:19.266 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.264+0000 7f1fc7fff640 1 -- 172.21.15.97:0/2151965549 >> v1:172.21.15.98:6789/0 conn(0x7f1fe013c9f0 legacy=0x7f1fe01dab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:19.267 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.264+0000 7f1fc7fff640 10 monclient: _finish_hunting 0 2024-03-25T18:20:19.267 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.264+0000 7f1fc7fff640 1 monclient: found mon.a 2024-03-25T18:20:19.267 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.264+0000 7f1fc7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:19.267 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.264+0000 7f1fc7fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:20:19.267 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.264+0000 7f1fc7fff640 1 -- 172.21.15.97:0/2151965549 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fe00b2b70 con 0x7f1fe0072af0 2024-03-25T18:20:19.267 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.264+0000 7f1fc7fff640 10 monclient: _finish_auth 0 2024-03-25T18:20:19.267 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.264+0000 7f1fc7fff640 10 monclient: _check_auth_tickets 2024-03-25T18:20:19.267 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.264+0000 7f1fc7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:19.267 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.264+0000 7f1fe604a640 5 monclient: authenticate success, global_id 4404 2024-03-25T18:20:19.267 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.264+0000 7f1fe604a640 10 monclient: _renew_subs 2024-03-25T18:20:19.267 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.264+0000 7f1fe604a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:20:19.267 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.264+0000 7f1fe604a640 1 -- 172.21.15.97:0/2151965549 --> v1:172.21.15.97:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fe01db270 con 0x7f1fe0072af0 2024-03-25T18:20:19.267 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.264+0000 7f1fe604a640 10 client.4404.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:20:19.268 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.264+0000 7f1fe604a640 10 monclient: _renew_subs 2024-03-25T18:20:19.268 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.264+0000 7f1fe604a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:20:19.268 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.264+0000 7f1fe604a640 1 -- 172.21.15.97:0/2151965549 --> v1:172.21.15.97:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fe0130640 con 0x7f1fe0072af0 2024-03-25T18:20:19.268 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.264+0000 7f1fe604a640 1 librados: init done 2024-03-25T18:20:19.268 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.265+0000 7f1fe604a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:19.268 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.265+0000 7f1fe604a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:19.268 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.265+0000 7f1fe604a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:20:19.268 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.265+0000 7f1fe604a640 1 -- 172.21.15.97:0/2151965549 --> v1:172.21.15.97:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fe00627b0 con 0x7f1fe0072af0 2024-03-25T18:20:19.556 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.555+0000 7f1fc7fff640 1 -- 172.21.15.97:0/2151965549 <== mon.0 v1:172.21.15.97:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1fd4001c50 con 0x7f1fe0072af0 2024-03-25T18:20:19.557 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.555+0000 7f1fc7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:19.557 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.555+0000 7f1fc7fff640 1 -- 172.21.15.97:0/2151965549 <== mon.0 v1:172.21.15.97:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f1fd4003a60 con 0x7f1fe0072af0 2024-03-25T18:20:19.557 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.555+0000 7f1fc7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:19.557 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.555+0000 7f1fc7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:20:19.557 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.555+0000 7f1fc7fff640 10 monclient: dump: 2024-03-25T18:20:19.557 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:20:19.557 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:20:19.557 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:20:19.558 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:20:19.558 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:19.558 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:20:19.558 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:20:19.558 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:20:19.558 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:20:19.558 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:20:19.558 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.555+0000 7f1fddffb640 4 set_mon_vals no callback set 2024-03-25T18:20:19.559 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.556+0000 7f1fc7fff640 1 -- 172.21.15.97:0/2151965549 <== mon.0 v1:172.21.15.97:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2377622137 0 0) 0x7f1fd4003c80 con 0x7f1fe0072af0 2024-03-25T18:20:19.559 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.557+0000 7f1fdf7fe640 10 monclient: get_auth_request con 0x7f1fcc06e050 auth_method 0 2024-03-25T18:20:19.559 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.557+0000 7f1fc7fff640 10 client.4404.objecter ms_handle_connect 0x7f1fcc06e050 2024-03-25T18:20:19.565 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.563+0000 7f1fc7fff640 1 -- 172.21.15.97:0/2151965549 <== mon.0 v1:172.21.15.97:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 430711918 0 0) 0x7f1fd40548b0 con 0x7f1fe0072af0 2024-03-25T18:20:19.565 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.563+0000 7f1fc7fff640 10 client.4404.objecter ms_dispatch 0x7f1fe0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:20:19.565 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.563+0000 7f1fc7fff640 3 client.4404.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:20:19.565 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.563+0000 7f1fc7fff640 3 client.4404.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:20:19.565 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.564+0000 7f1fc7fff640 20 client.4404.objecter dump_active .. 0 homeless 2024-03-25T18:20:19.565 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.564+0000 7f1fc7fff640 1 -- 172.21.15.97:0/2151965549 <== mon.0 v1:172.21.15.97:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1fd40503c0 con 0x7f1fe0072af0 2024-03-25T18:20:19.566 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.564+0000 7f1fc7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:19.566 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.564+0000 7f1fc7fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:20:19.811 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.809+0000 7f1fe604a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:19.811 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.809+0000 7f1fe604a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:19.811 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.809+0000 7f1fe604a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:20:19.811 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.809+0000 7f1fe604a640 1 -- 172.21.15.97:0/2151965549 --> v1:172.21.15.97:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1fe0069f40 con 0x7f1fe0072af0 2024-03-25T18:20:19.835 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.834+0000 7f1fc7fff640 1 -- 172.21.15.97:0/2151965549 <== mon.0 v1:172.21.15.97:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2169427412) 0x7f1fd40522f0 con 0x7f1fe0072af0 2024-03-25T18:20:19.836 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.834+0000 7f1fc7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:19.836 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.834+0000 7f1fc7fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:20:19.836 INFO:teuthology.misc.health.smithi097.stdout: 2024-03-25T18:20:19.836 INFO:teuthology.misc.health.smithi097.stdout:{"epoch":10,"fsid":"4c3d472c-79b8-486b-9e3d-2738d4306754","created":"2024-03-25T18:16:14.737309+0000","modified":"2024-03-25T18:16:20.357492+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:20.357492+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cb5152b9-262f-45c3-94b8-18d0e81df558","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6daf8793-0838-473d-a7d6-9a0d2894c75d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"136f1493-3b9c-4f43-a2e7-641482d69dae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"397d538c-4f64-4daf-9e8b-662d09f91327","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4f1df08-663f-4bda-93f8-3878ab7c24b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c4b18d70-a3ce-4b15-9b70-eae5525bd761","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8dd4033a-de0e-4f8a-96dc-59f3c1f825b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0077722e-ffa6-4a3d-9891-929d9869b286","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:20:19.838 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.836+0000 7f1fe604a640 10 librados: watch_flush enter 2024-03-25T18:20:19.838 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.836+0000 7f1fe604a640 10 librados: watch_flush exit 2024-03-25T18:20:19.838 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.836+0000 7f1fe604a640 20 client.4404.objecter shutdown clearing up homeless session... 2024-03-25T18:20:19.838 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.836+0000 7f1fe604a640 10 client.4404.objecter successfully canceled tick 2024-03-25T18:20:19.838 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.836+0000 7f1fe604a640 1 -- 172.21.15.97:0/2151965549 >> v1:172.21.15.98:6800/800387573 conn(0x7f1fcc06e050 legacy=0x7f1fcc070510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:19.838 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.836+0000 7f1fe604a640 10 monclient: shutdown 2024-03-25T18:20:19.838 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.836+0000 7f1fe604a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:19.839 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.836+0000 7f1fe604a640 1 -- 172.21.15.97:0/2151965549 >> v1:172.21.15.97:6789/0 conn(0x7f1fe0072af0 legacy=0x7f1fe0133110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:19.839 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.836+0000 7f1fe604a640 1 -- 172.21.15.97:0/2151965549 shutdown_connections 2024-03-25T18:20:19.839 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.836+0000 7f1fe604a640 1 -- 172.21.15.97:0/2151965549 >> 172.21.15.97:0/2151965549 conn(0x7f1fe00b0ef0 msgr2=0x7f1fe00b12d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:19.839 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.836+0000 7f1fe604a640 1 -- 172.21.15.97:0/2151965549 shutdown_connections 2024-03-25T18:20:19.839 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.836+0000 7f1fe604a640 1 -- 172.21.15.97:0/2151965549 wait complete. 2024-03-25T18:20:19.839 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:19.836+0000 7f1fe604a640 1 librados: shutdown 2024-03-25T18:20:19.848 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:20:26.654 DEBUG:teuthology.orchestra.run.smithi097:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:20:26.766 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.765+0000 7faec76e6640 10 monclient: get_monmap_and_config 2024-03-25T18:20:26.767 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.765+0000 7faec76e6640 10 monclient: build_initial_monmap 2024-03-25T18:20:26.767 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.765+0000 7faec76e6640 10 monclient: monmap: 2024-03-25T18:20:26.767 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:20:26.767 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:20:26.767 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:20:26.766778+0000 2024-03-25T18:20:26.767 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:20:26.766778+0000 2024-03-25T18:20:26.767 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:26.767 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:20:26.767 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:20:26.767 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T18:20:26.767 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-b 2024-03-25T18:20:26.767 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:20:26.768 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.766+0000 7faec76e6640 1 Processor -- start 2024-03-25T18:20:26.768 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.766+0000 7faec76e6640 1 -- start start 2024-03-25T18:20:26.768 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.766+0000 7faec76e6640 10 monclient: init 2024-03-25T18:20:26.768 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.767+0000 7faec76e6640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:26.769 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.767+0000 7faec76e6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:20:26.769 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.767+0000 7faec76e6640 10 monclient(hunting): picked mon.noname-a con 0x7faec00a98f0 addr v1:172.21.15.97:6789/0 2024-03-25T18:20:26.769 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.767+0000 7faec76e6640 10 monclient(hunting): picked mon.noname-b con 0x7faec013f710 addr v1:172.21.15.97:6790/0 2024-03-25T18:20:26.769 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.767+0000 7faec76e6640 10 monclient(hunting): picked mon.noname-c con 0x7faec01430c0 addr v1:172.21.15.98:6789/0 2024-03-25T18:20:26.769 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.767+0000 7faec76e6640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faec0142520 con 0x7faec00a98f0 2024-03-25T18:20:26.769 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.767+0000 7faec76e6640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faec00b28a0 con 0x7faec01430c0 2024-03-25T18:20:26.769 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.767+0000 7faec76e6640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faec00b2a30 con 0x7faec013f710 2024-03-25T18:20:26.769 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.767+0000 7faec76e6640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:26.769 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.767+0000 7faec76e6640 10 monclient(hunting): authenticate will time out at 7839.043135s 2024-03-25T18:20:26.779 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.778+0000 7faec5c83640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7faec01430c0 0x7faec014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.97:56732/0 (socket says 172.21.15.97:56732) 2024-03-25T18:20:26.779 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.778+0000 7faec5c83640 1 -- 172.21.15.97:0/2880826134 learned_addr learned my addr 172.21.15.97:0/2880826134 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:20:26.794 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.793+0000 7faeb7fff640 1 -- 172.21.15.97:0/2880826134 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 229817671 0 0) 0x7faec00b28a0 con 0x7faec01430c0 2024-03-25T18:20:26.794 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.793+0000 7faeb7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:20:26.794 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.793+0000 7faeb7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:26.794 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.793+0000 7faeb7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:26.794 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.793+0000 7faeb7fff640 10 monclient(hunting): my global_id is 4405 2024-03-25T18:20:26.794 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.793+0000 7faeb7fff640 1 -- 172.21.15.97:0/2880826134 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faeac003520 con 0x7faec01430c0 2024-03-25T18:20:26.800 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.798+0000 7faeb7fff640 1 -- 172.21.15.97:0/2880826134 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 37594203 0 0) 0x7faec0142520 con 0x7faec00a98f0 2024-03-25T18:20:26.800 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.798+0000 7faeb7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:20:26.800 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.798+0000 7faeb7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:26.800 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.798+0000 7faeb7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:26.800 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.798+0000 7faeb7fff640 10 monclient(hunting): my global_id is 4407 2024-03-25T18:20:26.800 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.798+0000 7faeb7fff640 1 -- 172.21.15.97:0/2880826134 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faec00b28a0 con 0x7faec00a98f0 2024-03-25T18:20:26.804 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.803+0000 7faeb7fff640 1 -- 172.21.15.97:0/2880826134 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 261901819 0 0) 0x7faec00b2a30 con 0x7faec013f710 2024-03-25T18:20:26.804 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.803+0000 7faeb7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:20:26.804 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.803+0000 7faeb7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:26.804 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.803+0000 7faeb7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:26.804 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.803+0000 7faeb7fff640 10 monclient(hunting): my global_id is 4397 2024-03-25T18:20:26.804 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.803+0000 7faeb7fff640 1 -- 172.21.15.97:0/2880826134 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faec0142520 con 0x7faec013f710 2024-03-25T18:20:26.811 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.810+0000 7faeb7fff640 1 -- 172.21.15.97:0/2880826134 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1377462461 0 0) 0x7faec00b28a0 con 0x7faec00a98f0 2024-03-25T18:20:26.811 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.810+0000 7faeb7fff640 1 -- 172.21.15.97:0/2880826134 >> v1:172.21.15.97:6790/0 conn(0x7faec013f710 legacy=0x7faec01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:26.812 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.810+0000 7faeb7fff640 1 -- 172.21.15.97:0/2880826134 >> v1:172.21.15.98:6789/0 conn(0x7faec01430c0 legacy=0x7faec014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:26.812 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.810+0000 7faeb7fff640 10 monclient: _finish_hunting 0 2024-03-25T18:20:26.812 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.810+0000 7faeb7fff640 1 monclient: found mon.noname-a 2024-03-25T18:20:26.812 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.810+0000 7faeb7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:26.812 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.810+0000 7faeb7fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.97:6789/0 2024-03-25T18:20:26.812 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.810+0000 7faeb7fff640 1 -- 172.21.15.97:0/2880826134 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faec00b44c0 con 0x7faec00a98f0 2024-03-25T18:20:26.812 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.810+0000 7faeb7fff640 10 monclient: _finish_auth 0 2024-03-25T18:20:26.812 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.810+0000 7faeb7fff640 10 monclient: _check_auth_tickets 2024-03-25T18:20:26.812 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.810+0000 7faeb7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:26.812 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.810+0000 7faeb7fff640 1 -- 172.21.15.97:0/2880826134 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7faebc001670 con 0x7faec00a98f0 2024-03-25T18:20:26.812 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.810+0000 7faec76e6640 5 monclient: authenticate success, global_id 4407 2024-03-25T18:20:26.812 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.810+0000 7faeb7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:26.812 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.810+0000 7faeb7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:20:26.812 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.810+0000 7faeb7fff640 10 monclient: dump: 2024-03-25T18:20:26.812 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:20:26.813 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:20:26.813 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:20:26.813 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:20:26.813 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:26.813 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:20:26.813 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:20:26.813 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:20:26.813 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:20:26.813 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:20:26.813 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.810+0000 7faec76e6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:20:26.827 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.825+0000 7faeb7fff640 1 -- 172.21.15.97:0/2880826134 <== mon.0 v1:172.21.15.97:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faebc0017d0 con 0x7faec00a98f0 2024-03-25T18:20:26.827 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.825+0000 7faeb7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:26.827 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.825+0000 7faeb7fff640 1 -- 172.21.15.97:0/2880826134 <== mon.0 v1:172.21.15.97:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7faebc001be0 con 0x7faec00a98f0 2024-03-25T18:20:26.827 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.825+0000 7faeb7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:26.827 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.826+0000 7faeb7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:20:26.827 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.826+0000 7faeb7fff640 10 monclient: dump: 2024-03-25T18:20:26.827 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:20:26.827 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:20:26.827 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:20:26.827 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:20:26.827 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:26.827 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:20:26.827 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:20:26.828 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:20:26.828 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:20:26.828 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:20:26.828 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.826+0000 7faec76e6640 10 monclient: get_monmap_and_config success 2024-03-25T18:20:26.828 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.826+0000 7faec76e6640 4 set_mon_vals no callback set 2024-03-25T18:20:26.830 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.826+0000 7faec76e6640 10 monclient: shutdown 2024-03-25T18:20:26.830 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.826+0000 7faec76e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:26.830 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.826+0000 7faec76e6640 1 -- 172.21.15.97:0/2880826134 >> v1:172.21.15.97:6789/0 conn(0x7faec00a98f0 legacy=0x7faec00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:26.830 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.826+0000 7faec76e6640 1 -- 172.21.15.97:0/2880826134 shutdown_connections 2024-03-25T18:20:26.830 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.826+0000 7faec76e6640 1 -- 172.21.15.97:0/2880826134 >> 172.21.15.97:0/2880826134 conn(0x7faec00b7ea0 msgr2=0x7faec00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:26.830 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.826+0000 7faec76e6640 1 -- 172.21.15.97:0/2880826134 shutdown_connections 2024-03-25T18:20:26.830 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.826+0000 7faec76e6640 1 -- 172.21.15.97:0/2880826134 wait complete. 2024-03-25T18:20:26.830 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.827+0000 7faec76e6640 10 monclient: build_initial_monmap 2024-03-25T18:20:26.830 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.827+0000 7faec76e6640 10 monclient: monmap: 2024-03-25T18:20:26.830 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:20:26.830 INFO:teuthology.misc.health.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:20:26.831 INFO:teuthology.misc.health.smithi097.stderr:last_changed 0.000000 2024-03-25T18:20:26.831 INFO:teuthology.misc.health.smithi097.stderr:created 0.000000 2024-03-25T18:20:26.831 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:26.831 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 1 2024-03-25T18:20:26.831 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:20:26.831 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T18:20:26.831 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-c 2024-03-25T18:20:26.831 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:20:26.831 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.827+0000 7faec76e6640 1 librados: starting msgr at 2024-03-25T18:20:26.831 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.827+0000 7faec76e6640 1 librados: starting objecter 2024-03-25T18:20:26.831 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.827+0000 7faec76e6640 1 Processor -- start 2024-03-25T18:20:26.831 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.828+0000 7faec76e6640 1 -- start start 2024-03-25T18:20:26.831 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.828+0000 7faec76e6640 1 librados: setting wanted keys 2024-03-25T18:20:26.831 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.828+0000 7faec76e6640 1 librados: calling monclient init 2024-03-25T18:20:26.831 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.828+0000 7faec76e6640 10 monclient: init 2024-03-25T18:20:26.831 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.828+0000 7faec76e6640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:26.832 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.828+0000 7faec76e6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:20:26.832 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.828+0000 7faec76e6640 10 monclient(hunting): picked mon.noname-b con 0x7faec00a98f0 addr v1:172.21.15.98:6789/0 2024-03-25T18:20:26.832 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.828+0000 7faec76e6640 10 monclient(hunting): picked mon.noname-c con 0x7faec013f710 addr v1:172.21.15.97:6790/0 2024-03-25T18:20:26.832 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.828+0000 7faec76e6640 10 monclient(hunting): picked mon.noname-a con 0x7faec01430c0 addr v1:172.21.15.97:6789/0 2024-03-25T18:20:26.832 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.828+0000 7faec76e6640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faec0138f10 con 0x7faec01430c0 2024-03-25T18:20:26.832 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.828+0000 7faec76e6640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faec0139970 con 0x7faec00a98f0 2024-03-25T18:20:26.832 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.828+0000 7faec76e6640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faec00abb50 con 0x7faec013f710 2024-03-25T18:20:26.832 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.828+0000 7faec76e6640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:26.832 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.828+0000 7faec76e6640 10 monclient(hunting): authenticate will time out at 7839.104534s 2024-03-25T18:20:26.841 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.840+0000 7faec5482640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7faec00a98f0 0x7faec0136330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.97:56768/0 (socket says 172.21.15.97:56768) 2024-03-25T18:20:26.842 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.840+0000 7faec5c83640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7faec01430c0 0x7faec0138320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:47864/0 (socket says 172.21.15.97:47864) 2024-03-25T18:20:26.842 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.840+0000 7faec5482640 1 -- 172.21.15.97:0/3415502866 learned_addr learned my addr 172.21.15.97:0/3415502866 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:20:26.842 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.840+0000 7faeb5ffb640 10 client.?.objecter ms_handle_connect 0x7faec01430c0 2024-03-25T18:20:26.842 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.840+0000 7faeb5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:26.844 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.843+0000 7faeb5ffb640 10 client.?.objecter ms_handle_connect 0x7faec00a98f0 2024-03-25T18:20:26.844 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.843+0000 7faeb5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:26.850 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.849+0000 7faeb5ffb640 1 -- 172.21.15.97:0/3415502866 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2009952215 0 0) 0x7faec0139970 con 0x7faec00a98f0 2024-03-25T18:20:26.850 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.849+0000 7faeb5ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:20:26.850 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.849+0000 7faeb5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:26.850 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.849+0000 7faeb5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:26.850 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.849+0000 7faeb5ffb640 10 monclient(hunting): my global_id is 4408 2024-03-25T18:20:26.850 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.849+0000 7faeb5ffb640 1 -- 172.21.15.97:0/3415502866 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faea4004eb0 con 0x7faec00a98f0 2024-03-25T18:20:26.855 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.853+0000 7faeb5ffb640 1 -- 172.21.15.97:0/3415502866 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3215778659 0 0) 0x7faec0138f10 con 0x7faec01430c0 2024-03-25T18:20:26.855 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.853+0000 7faeb5ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:20:26.855 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.853+0000 7faeb5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:26.855 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.853+0000 7faeb5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:26.855 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.853+0000 7faeb5ffb640 10 monclient(hunting): my global_id is 4410 2024-03-25T18:20:26.855 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.853+0000 7faeb5ffb640 1 -- 172.21.15.97:0/3415502866 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faec0139970 con 0x7faec01430c0 2024-03-25T18:20:26.862 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.861+0000 7faeb5ffb640 10 client.?.objecter ms_handle_connect 0x7faec013f710 2024-03-25T18:20:26.862 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.861+0000 7faeb5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:26.865 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.864+0000 7faeb5ffb640 1 -- 172.21.15.97:0/3415502866 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 260716551 0 0) 0x7faea4004eb0 con 0x7faec00a98f0 2024-03-25T18:20:26.866 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.864+0000 7faeb5ffb640 1 -- 172.21.15.97:0/3415502866 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faec0138f10 con 0x7faec00a98f0 2024-03-25T18:20:26.866 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.864+0000 7faeb5ffb640 1 -- 172.21.15.97:0/3415502866 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7faebc001670 con 0x7faec00a98f0 2024-03-25T18:20:26.866 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.864+0000 7faeb5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:26.866 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.864+0000 7faeb5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:20:26.866 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.864+0000 7faeb5ffb640 10 monclient(hunting): dump: 2024-03-25T18:20:26.866 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:20:26.866 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:20:26.866 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:20:26.866 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:20:26.866 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:26.866 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:20:26.866 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:20:26.866 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:20:26.866 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:20:26.866 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:20:26.867 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.864+0000 7faeb5ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:20:26.870 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.869+0000 7faeb5ffb640 1 -- 172.21.15.97:0/3415502866 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3033215536 0 0) 0x7faec0139970 con 0x7faec01430c0 2024-03-25T18:20:26.870 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.869+0000 7faeb5ffb640 1 -- 172.21.15.97:0/3415502866 --> v1:172.21.15.97:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faea4004eb0 con 0x7faec01430c0 2024-03-25T18:20:26.870 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.869+0000 7faeb5ffb640 1 -- 172.21.15.97:0/3415502866 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7faeb8001af0 con 0x7faec01430c0 2024-03-25T18:20:26.870 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.869+0000 7faeb5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:26.870 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.869+0000 7faeb5ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:20:26.870 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.869+0000 7faeb5ffb640 10 monclient(hunting): dump: 2024-03-25T18:20:26.870 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:20:26.870 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:20:26.870 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:20:26.871 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:20:26.871 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:26.871 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:20:26.871 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:20:26.871 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:20:26.871 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:20:26.871 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:20:26.877 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.876+0000 7faeb5ffb640 1 -- 172.21.15.97:0/3415502866 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 266906098 0 0) 0x7faec00abb50 con 0x7faec013f710 2024-03-25T18:20:26.877 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.876+0000 7faeb5ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:20:26.877 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.876+0000 7faeb5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:26.878 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.876+0000 7faeb5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:26.878 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.876+0000 7faeb5ffb640 10 monclient(hunting): my global_id is 4400 2024-03-25T18:20:26.878 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.876+0000 7faeb5ffb640 1 -- 172.21.15.97:0/3415502866 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faec0139970 con 0x7faec013f710 2024-03-25T18:20:26.882 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.880+0000 7faeb5ffb640 1 -- 172.21.15.97:0/3415502866 <== mon.0 v1:172.21.15.97:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1660651755 0 0) 0x7faea4004eb0 con 0x7faec01430c0 2024-03-25T18:20:26.882 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.880+0000 7faeb5ffb640 1 -- 172.21.15.97:0/3415502866 >> v1:172.21.15.97:6790/0 conn(0x7faec013f710 legacy=0x7faec00ac250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:26.882 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.880+0000 7faeb5ffb640 1 -- 172.21.15.97:0/3415502866 >> v1:172.21.15.98:6789/0 conn(0x7faec00a98f0 legacy=0x7faec0136330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:26.882 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.880+0000 7faeb5ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:20:26.882 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.880+0000 7faeb5ffb640 1 monclient: found mon.a 2024-03-25T18:20:26.882 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.880+0000 7faeb5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:26.883 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.880+0000 7faeb5ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:20:26.883 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.880+0000 7faeb5ffb640 1 -- 172.21.15.97:0/3415502866 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faec00abe30 con 0x7faec01430c0 2024-03-25T18:20:26.883 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.880+0000 7faeb5ffb640 10 monclient: _finish_auth 0 2024-03-25T18:20:26.883 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.881+0000 7faeb5ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:20:26.883 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.881+0000 7faeb5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:26.883 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.881+0000 7faec76e6640 5 monclient: authenticate success, global_id 4410 2024-03-25T18:20:26.883 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.881+0000 7faec76e6640 10 monclient: _renew_subs 2024-03-25T18:20:26.883 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.881+0000 7faec76e6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:20:26.883 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.881+0000 7faec76e6640 1 -- 172.21.15.97:0/3415502866 --> v1:172.21.15.97:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faec0137b40 con 0x7faec01430c0 2024-03-25T18:20:26.883 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.881+0000 7faec76e6640 10 client.4410.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:20:26.883 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.881+0000 7faec76e6640 10 monclient: _renew_subs 2024-03-25T18:20:26.883 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.881+0000 7faec76e6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:20:26.883 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.881+0000 7faec76e6640 1 -- 172.21.15.97:0/3415502866 --> v1:172.21.15.97:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7faec0138c10 con 0x7faec01430c0 2024-03-25T18:20:26.883 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.881+0000 7faec76e6640 1 librados: init done 2024-03-25T18:20:26.884 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.882+0000 7faec76e6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:26.884 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.882+0000 7faec76e6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:26.884 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.882+0000 7faec76e6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:20:26.884 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:26.882+0000 7faec76e6640 1 -- 172.21.15.97:0/3415502866 --> v1:172.21.15.97:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faec00627b0 con 0x7faec01430c0 2024-03-25T18:20:27.183 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:27.181+0000 7faeb5ffb640 1 -- 172.21.15.97:0/3415502866 <== mon.0 v1:172.21.15.97:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faeb8001f00 con 0x7faec01430c0 2024-03-25T18:20:27.183 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:27.181+0000 7faeb5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:27.183 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:27.181+0000 7faeb5ffb640 1 -- 172.21.15.97:0/3415502866 <== mon.0 v1:172.21.15.97:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7faeb8004b60 con 0x7faec01430c0 2024-03-25T18:20:27.183 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:27.181+0000 7faeb5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:27.183 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:27.181+0000 7faeb5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:20:27.183 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:27.181+0000 7faeb5ffb640 10 monclient: dump: 2024-03-25T18:20:27.183 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:20:27.183 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:20:27.183 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:20:27.184 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:20:27.184 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:27.184 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:20:27.184 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:20:27.184 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:20:27.184 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:20:27.184 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:20:27.184 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:27.181+0000 7faeb77fe640 4 set_mon_vals no callback set 2024-03-25T18:20:27.184 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:27.182+0000 7faeb5ffb640 1 -- 172.21.15.97:0/3415502866 <== mon.0 v1:172.21.15.97:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2377622137 0 0) 0x7faeb8004d80 con 0x7faec01430c0 2024-03-25T18:20:27.185 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:27.183+0000 7faec5482640 10 monclient: get_auth_request con 0x7faea406e0f0 auth_method 0 2024-03-25T18:20:27.185 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:27.183+0000 7faeb5ffb640 10 client.4410.objecter ms_handle_connect 0x7faea406e0f0 2024-03-25T18:20:27.190 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:27.188+0000 7faeb5ffb640 1 -- 172.21.15.97:0/3415502866 <== mon.0 v1:172.21.15.97:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 430711918 0 0) 0x7faeb8054990 con 0x7faec01430c0 2024-03-25T18:20:27.190 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:27.188+0000 7faeb5ffb640 10 client.4410.objecter ms_dispatch 0x7faec0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:20:27.190 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:27.188+0000 7faeb5ffb640 3 client.4410.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:20:27.190 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:27.188+0000 7faeb5ffb640 3 client.4410.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:20:27.190 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:27.189+0000 7faeb5ffb640 20 client.4410.objecter dump_active .. 0 homeless 2024-03-25T18:20:27.190 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:27.189+0000 7faeb5ffb640 1 -- 172.21.15.97:0/3415502866 <== mon.0 v1:172.21.15.97:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7faeb80502d0 con 0x7faec01430c0 2024-03-25T18:20:27.190 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:27.189+0000 7faeb5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:27.190 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:27.189+0000 7faeb5ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:20:27.437 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:27.435+0000 7faec76e6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:27.437 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:27.435+0000 7faec76e6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:27.437 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:27.435+0000 7faec76e6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:20:27.437 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:27.435+0000 7faec76e6640 1 -- 172.21.15.97:0/3415502866 --> v1:172.21.15.97:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faec013db40 con 0x7faec01430c0 2024-03-25T18:20:27.459 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:27.457+0000 7faeb5ffb640 1 -- 172.21.15.97:0/3415502866 <== mon.0 v1:172.21.15.97:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2169427412) 0x7faeb80523d0 con 0x7faec01430c0 2024-03-25T18:20:27.459 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:27.457+0000 7faeb5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:27.459 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:27.457+0000 7faeb5ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:20:27.459 INFO:teuthology.misc.health.smithi097.stdout: 2024-03-25T18:20:27.459 INFO:teuthology.misc.health.smithi097.stdout:{"epoch":10,"fsid":"4c3d472c-79b8-486b-9e3d-2738d4306754","created":"2024-03-25T18:16:14.737309+0000","modified":"2024-03-25T18:16:20.357492+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:20.357492+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cb5152b9-262f-45c3-94b8-18d0e81df558","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6daf8793-0838-473d-a7d6-9a0d2894c75d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"136f1493-3b9c-4f43-a2e7-641482d69dae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"397d538c-4f64-4daf-9e8b-662d09f91327","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4f1df08-663f-4bda-93f8-3878ab7c24b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c4b18d70-a3ce-4b15-9b70-eae5525bd761","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8dd4033a-de0e-4f8a-96dc-59f3c1f825b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0077722e-ffa6-4a3d-9891-929d9869b286","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:20:27.461 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:27.459+0000 7faec76e6640 10 librados: watch_flush enter 2024-03-25T18:20:27.461 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:27.459+0000 7faec76e6640 10 librados: watch_flush exit 2024-03-25T18:20:27.461 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:27.459+0000 7faec76e6640 20 client.4410.objecter shutdown clearing up homeless session... 2024-03-25T18:20:27.461 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:27.459+0000 7faec76e6640 10 client.4410.objecter successfully canceled tick 2024-03-25T18:20:27.462 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:27.460+0000 7faec76e6640 1 -- 172.21.15.97:0/3415502866 >> v1:172.21.15.98:6800/800387573 conn(0x7faea406e0f0 legacy=0x7faea40705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:27.462 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:27.460+0000 7faec76e6640 10 monclient: shutdown 2024-03-25T18:20:27.462 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:27.460+0000 7faec76e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:27.462 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:27.460+0000 7faec76e6640 1 -- 172.21.15.97:0/3415502866 >> v1:172.21.15.97:6789/0 conn(0x7faec01430c0 legacy=0x7faec0138320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:27.462 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:27.460+0000 7faec76e6640 1 -- 172.21.15.97:0/3415502866 shutdown_connections 2024-03-25T18:20:27.462 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:27.460+0000 7faec76e6640 1 -- 172.21.15.97:0/3415502866 >> 172.21.15.97:0/3415502866 conn(0x7faec00b7ea0 msgr2=0x7faec0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:27.462 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:27.460+0000 7faec76e6640 1 -- 172.21.15.97:0/3415502866 shutdown_connections 2024-03-25T18:20:27.462 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:27.460+0000 7faec76e6640 1 -- 172.21.15.97:0/3415502866 wait complete. 2024-03-25T18:20:27.462 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:27.460+0000 7faec76e6640 1 librados: shutdown 2024-03-25T18:20:27.472 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:20:32.558 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:04:34.597 34931== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:32.615 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:04:34.653 34931== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:32.645 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:04:34.684 34931== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:32.678 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:04:34.721 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:32.684 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:04:34.723 34935== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:32.688 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:04:34.731 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:32.697 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:04:34.735 34935== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:32.710 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:04:34.754 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:32.733 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:04:34.772 34935== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:34.279 DEBUG:teuthology.orchestra.run.smithi097:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:20:34.396 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.395+0000 7f495c493640 10 monclient: get_monmap_and_config 2024-03-25T18:20:34.396 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.395+0000 7f495c493640 10 monclient: build_initial_monmap 2024-03-25T18:20:34.396 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.395+0000 7f495c493640 10 monclient: monmap: 2024-03-25T18:20:34.396 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:20:34.396 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:20:34.396 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:20:34.396055+0000 2024-03-25T18:20:34.397 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:20:34.396055+0000 2024-03-25T18:20:34.397 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:34.397 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:20:34.397 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:20:34.397 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T18:20:34.397 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-b 2024-03-25T18:20:34.397 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:20:34.399 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.396+0000 7f495c493640 1 Processor -- start 2024-03-25T18:20:34.399 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.396+0000 7f495c493640 1 -- start start 2024-03-25T18:20:34.399 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.396+0000 7f495c493640 10 monclient: init 2024-03-25T18:20:34.399 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.397+0000 7f495c493640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:34.399 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.397+0000 7f495c493640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:20:34.399 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.397+0000 7f495c493640 10 monclient(hunting): picked mon.noname-b con 0x7f49540a98f0 addr v1:172.21.15.97:6790/0 2024-03-25T18:20:34.400 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.397+0000 7f495c493640 10 monclient(hunting): picked mon.noname-c con 0x7f495413f710 addr v1:172.21.15.98:6789/0 2024-03-25T18:20:34.400 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.397+0000 7f495c493640 10 monclient(hunting): picked mon.noname-a con 0x7f49541430c0 addr v1:172.21.15.97:6789/0 2024-03-25T18:20:34.400 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.397+0000 7f495c493640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4954142520 con 0x7f49541430c0 2024-03-25T18:20:34.400 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.397+0000 7f495c493640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49540b28a0 con 0x7f495413f710 2024-03-25T18:20:34.400 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.397+0000 7f495c493640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49540b2a30 con 0x7f49540a98f0 2024-03-25T18:20:34.400 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.397+0000 7f495c493640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:34.400 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.398+0000 7f495c493640 10 monclient(hunting): authenticate will time out at 7846.672902s 2024-03-25T18:20:34.410 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.410+0000 7f495a22f640 1 --1- >> v1:172.21.15.97:6790/0 conn(0x7f49540a98f0 0x7f49540a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6790/0 says I am v1:172.21.15.97:39220/0 (socket says 172.21.15.97:39220) 2024-03-25T18:20:34.410 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.410+0000 7f495a22f640 1 -- 172.21.15.97:0/1238178973 learned_addr learned my addr 172.21.15.97:0/1238178973 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:20:34.426 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.426+0000 7f495922d640 1 -- 172.21.15.97:0/1238178973 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2982638422 0 0) 0x7f49540b2a30 con 0x7f49540a98f0 2024-03-25T18:20:34.426 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.426+0000 7f495922d640 10 monclient(hunting): _negotiate 2024-03-25T18:20:34.426 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.426+0000 7f495922d640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:34.426 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.426+0000 7f495922d640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:34.426 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.426+0000 7f495922d640 10 monclient(hunting): my global_id is 4403 2024-03-25T18:20:34.426 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.426+0000 7f495922d640 1 -- 172.21.15.97:0/1238178973 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4940003520 con 0x7f49540a98f0 2024-03-25T18:20:34.434 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.434+0000 7f495922d640 1 -- 172.21.15.97:0/1238178973 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1245437618 0 0) 0x7f4954142520 con 0x7f49541430c0 2024-03-25T18:20:34.434 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.434+0000 7f495922d640 10 monclient(hunting): _negotiate 2024-03-25T18:20:34.434 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.434+0000 7f495922d640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:34.434 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.434+0000 7f495922d640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:34.435 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.434+0000 7f495922d640 10 monclient(hunting): my global_id is 4413 2024-03-25T18:20:34.435 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.434+0000 7f495922d640 1 -- 172.21.15.97:0/1238178973 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49540b2a30 con 0x7f49541430c0 2024-03-25T18:20:34.435 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.434+0000 7f495922d640 1 -- 172.21.15.97:0/1238178973 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4261463507 0 0) 0x7f49540b28a0 con 0x7f495413f710 2024-03-25T18:20:34.435 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.434+0000 7f495922d640 10 monclient(hunting): _negotiate 2024-03-25T18:20:34.435 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.434+0000 7f495922d640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:34.435 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.434+0000 7f495922d640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:34.435 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.434+0000 7f495922d640 10 monclient(hunting): my global_id is 4411 2024-03-25T18:20:34.435 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.434+0000 7f495922d640 1 -- 172.21.15.97:0/1238178973 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4954142520 con 0x7f495413f710 2024-03-25T18:20:34.443 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.442+0000 7f495922d640 1 -- 172.21.15.97:0/1238178973 <== mon.2 v1:172.21.15.97:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4000902841 0 0) 0x7f4940003520 con 0x7f49540a98f0 2024-03-25T18:20:34.443 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.442+0000 7f495922d640 1 -- 172.21.15.97:0/1238178973 >> v1:172.21.15.98:6789/0 conn(0x7f495413f710 legacy=0x7f49541429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:34.443 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.442+0000 7f495922d640 1 -- 172.21.15.97:0/1238178973 >> v1:172.21.15.97:6789/0 conn(0x7f49541430c0 legacy=0x7f495414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:34.443 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.442+0000 7f495922d640 10 monclient: _finish_hunting 0 2024-03-25T18:20:34.443 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.442+0000 7f495922d640 1 monclient: found mon.noname-b 2024-03-25T18:20:34.443 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.442+0000 7f495922d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:34.443 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.442+0000 7f495922d640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.97:6790/0 2024-03-25T18:20:34.443 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.442+0000 7f495922d640 1 -- 172.21.15.97:0/1238178973 --> v1:172.21.15.97:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49540b44c0 con 0x7f49540a98f0 2024-03-25T18:20:34.443 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.442+0000 7f495922d640 10 monclient: _finish_auth 0 2024-03-25T18:20:34.444 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.442+0000 7f495922d640 10 monclient: _check_auth_tickets 2024-03-25T18:20:34.444 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.442+0000 7f495922d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:34.444 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.442+0000 7f495922d640 1 -- 172.21.15.97:0/1238178973 <== mon.2 v1:172.21.15.97:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f4950001a10 con 0x7f49540a98f0 2024-03-25T18:20:34.444 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.442+0000 7f495c493640 5 monclient: authenticate success, global_id 4403 2024-03-25T18:20:34.444 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.442+0000 7f495c493640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:20:34.444 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.442+0000 7f495922d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:34.444 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.442+0000 7f495922d640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:20:34.444 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.442+0000 7f495922d640 10 monclient: dump: 2024-03-25T18:20:34.444 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:20:34.444 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:20:34.444 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:20:34.444 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:20:34.444 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:34.444 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:20:34.444 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:20:34.445 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:20:34.445 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:20:34.445 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:20:34.445 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.442+0000 7f495c493640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:20:34.448 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.448+0000 7f495922d640 1 -- 172.21.15.97:0/1238178973 <== mon.2 v1:172.21.15.97:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4950001b70 con 0x7f49540a98f0 2024-03-25T18:20:34.449 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.448+0000 7f495922d640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:34.449 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.448+0000 7f495922d640 1 -- 172.21.15.97:0/1238178973 <== mon.2 v1:172.21.15.97:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f4950001f80 con 0x7f49540a98f0 2024-03-25T18:20:34.449 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.448+0000 7f495922d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:34.449 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.448+0000 7f495922d640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:20:34.449 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.448+0000 7f495922d640 10 monclient: dump: 2024-03-25T18:20:34.449 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:20:34.449 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:20:34.449 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:20:34.449 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:20:34.449 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:34.449 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:20:34.449 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:20:34.449 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:20:34.450 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:20:34.450 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:20:34.450 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.448+0000 7f495c493640 10 monclient: get_monmap_and_config success 2024-03-25T18:20:34.450 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.448+0000 7f495c493640 4 set_mon_vals no callback set 2024-03-25T18:20:34.452 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.449+0000 7f495c493640 10 monclient: shutdown 2024-03-25T18:20:34.452 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.449+0000 7f495c493640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:34.452 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.449+0000 7f495c493640 1 -- 172.21.15.97:0/1238178973 >> v1:172.21.15.97:6790/0 conn(0x7f49540a98f0 legacy=0x7f49540a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:34.452 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.449+0000 7f495c493640 1 -- 172.21.15.97:0/1238178973 shutdown_connections 2024-03-25T18:20:34.452 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.449+0000 7f495c493640 1 -- 172.21.15.97:0/1238178973 >> 172.21.15.97:0/1238178973 conn(0x7f49540b7ea0 msgr2=0x7f49540b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:34.452 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.449+0000 7f495c493640 1 -- 172.21.15.97:0/1238178973 shutdown_connections 2024-03-25T18:20:34.452 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.449+0000 7f495c493640 1 -- 172.21.15.97:0/1238178973 wait complete. 2024-03-25T18:20:34.452 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.450+0000 7f495c493640 10 monclient: build_initial_monmap 2024-03-25T18:20:34.452 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.450+0000 7f495c493640 10 monclient: monmap: 2024-03-25T18:20:34.452 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:20:34.452 INFO:teuthology.misc.health.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:20:34.453 INFO:teuthology.misc.health.smithi097.stderr:last_changed 0.000000 2024-03-25T18:20:34.453 INFO:teuthology.misc.health.smithi097.stderr:created 0.000000 2024-03-25T18:20:34.453 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:34.453 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 1 2024-03-25T18:20:34.453 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:20:34.453 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T18:20:34.453 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-c 2024-03-25T18:20:34.453 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:20:34.453 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.450+0000 7f495c493640 1 librados: starting msgr at 2024-03-25T18:20:34.453 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.450+0000 7f495c493640 1 librados: starting objecter 2024-03-25T18:20:34.453 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.450+0000 7f495c493640 1 Processor -- start 2024-03-25T18:20:34.453 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.450+0000 7f495c493640 1 -- start start 2024-03-25T18:20:34.453 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.450+0000 7f495c493640 1 librados: setting wanted keys 2024-03-25T18:20:34.453 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.450+0000 7f495c493640 1 librados: calling monclient init 2024-03-25T18:20:34.454 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.450+0000 7f495c493640 10 monclient: init 2024-03-25T18:20:34.454 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.451+0000 7f495c493640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:34.454 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.451+0000 7f495c493640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:20:34.454 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.451+0000 7f495c493640 10 monclient(hunting): picked mon.noname-b con 0x7f49540a98f0 addr v1:172.21.15.98:6789/0 2024-03-25T18:20:34.454 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.451+0000 7f495c493640 10 monclient(hunting): picked mon.noname-c con 0x7f495413f710 addr v1:172.21.15.97:6790/0 2024-03-25T18:20:34.454 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.451+0000 7f495c493640 10 monclient(hunting): picked mon.noname-a con 0x7f49541430c0 addr v1:172.21.15.97:6789/0 2024-03-25T18:20:34.454 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.451+0000 7f495c493640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4954138b20 con 0x7f49541430c0 2024-03-25T18:20:34.454 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.451+0000 7f495c493640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4954137a80 con 0x7f49540a98f0 2024-03-25T18:20:34.454 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.451+0000 7f495c493640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4954139880 con 0x7f495413f710 2024-03-25T18:20:34.454 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.451+0000 7f495c493640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:34.454 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.451+0000 7f495c493640 10 monclient(hunting): authenticate will time out at 7846.726592s 2024-03-25T18:20:34.459 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.459+0000 7f495a22f640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f49540a98f0 0x7f4954137370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.97:54450/0 (socket says 172.21.15.97:54450) 2024-03-25T18:20:34.459 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.459+0000 7f495a22f640 1 -- 172.21.15.97:0/3257994277 learned_addr learned my addr 172.21.15.97:0/3257994277 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:20:34.459 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.459+0000 7f494affd640 10 client.?.objecter ms_handle_connect 0x7f49540a98f0 2024-03-25T18:20:34.459 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.459+0000 7f494affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:34.467 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.467+0000 7f494affd640 1 -- 172.21.15.97:0/3257994277 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4001785538 0 0) 0x7f4954137a80 con 0x7f49540a98f0 2024-03-25T18:20:34.468 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.467+0000 7f494affd640 10 monclient(hunting): _negotiate 2024-03-25T18:20:34.468 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.467+0000 7f494affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:34.468 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.467+0000 7f494affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:34.468 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.467+0000 7f494affd640 10 monclient(hunting): my global_id is 4414 2024-03-25T18:20:34.468 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.467+0000 7f494affd640 1 -- 172.21.15.97:0/3257994277 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4938004eb0 con 0x7f49540a98f0 2024-03-25T18:20:34.479 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.479+0000 7f494affd640 10 client.?.objecter ms_handle_connect 0x7f495413f710 2024-03-25T18:20:34.479 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.479+0000 7f494affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:34.479 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.479+0000 7f494affd640 10 client.?.objecter ms_handle_connect 0x7f49541430c0 2024-03-25T18:20:34.479 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.479+0000 7f494affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:34.483 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.482+0000 7f494affd640 1 -- 172.21.15.97:0/3257994277 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1942432415 0 0) 0x7f4938004eb0 con 0x7f49540a98f0 2024-03-25T18:20:34.483 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.482+0000 7f494affd640 1 -- 172.21.15.97:0/3257994277 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4954137a80 con 0x7f49540a98f0 2024-03-25T18:20:34.483 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.482+0000 7f494affd640 1 -- 172.21.15.97:0/3257994277 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f4950001fc0 con 0x7f49540a98f0 2024-03-25T18:20:34.483 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.482+0000 7f494affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:34.483 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.482+0000 7f494affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:20:34.483 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.482+0000 7f494affd640 10 monclient(hunting): dump: 2024-03-25T18:20:34.483 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:20:34.483 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:20:34.483 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:20:34.483 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:20:34.484 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:34.484 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:20:34.484 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:20:34.484 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:20:34.484 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:20:34.484 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:20:34.484 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.482+0000 7f494affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:20:34.492 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.492+0000 7f494affd640 1 -- 172.21.15.97:0/3257994277 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2308464150 0 0) 0x7f4954139880 con 0x7f495413f710 2024-03-25T18:20:34.493 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.492+0000 7f494affd640 10 monclient(hunting): _negotiate 2024-03-25T18:20:34.493 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.492+0000 7f494affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:34.493 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.492+0000 7f494affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:34.493 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.492+0000 7f494affd640 10 monclient(hunting): my global_id is 4406 2024-03-25T18:20:34.493 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.492+0000 7f494affd640 1 -- 172.21.15.97:0/3257994277 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4938004eb0 con 0x7f495413f710 2024-03-25T18:20:34.504 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.503+0000 7f494affd640 1 -- 172.21.15.97:0/3257994277 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3192431536 0 0) 0x7f4954138b20 con 0x7f49541430c0 2024-03-25T18:20:34.504 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.503+0000 7f494affd640 10 monclient(hunting): _negotiate 2024-03-25T18:20:34.504 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.503+0000 7f494affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:34.504 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.503+0000 7f494affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:34.504 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.503+0000 7f494affd640 10 monclient(hunting): my global_id is 4416 2024-03-25T18:20:34.504 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.504+0000 7f494affd640 1 -- 172.21.15.97:0/3257994277 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4954139880 con 0x7f49541430c0 2024-03-25T18:20:34.507 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.507+0000 7f494affd640 1 -- 172.21.15.97:0/3257994277 <== mon.1 v1:172.21.15.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3341188157 0 0) 0x7f4954137a80 con 0x7f49540a98f0 2024-03-25T18:20:34.508 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.507+0000 7f494affd640 1 -- 172.21.15.97:0/3257994277 >> v1:172.21.15.97:6790/0 conn(0x7f495413f710 legacy=0x7f4954139170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:34.508 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.507+0000 7f494affd640 1 -- 172.21.15.97:0/3257994277 >> v1:172.21.15.97:6789/0 conn(0x7f49541430c0 legacy=0x7f49541e2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:34.508 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.507+0000 7f494affd640 10 monclient: _finish_hunting 0 2024-03-25T18:20:34.508 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.507+0000 7f494affd640 1 monclient: found mon.b 2024-03-25T18:20:34.508 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.507+0000 7f494affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:34.508 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.507+0000 7f494affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:20:34.508 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.507+0000 7f494affd640 1 -- 172.21.15.97:0/3257994277 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4954139a10 con 0x7f49540a98f0 2024-03-25T18:20:34.508 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.507+0000 7f494affd640 10 monclient: _finish_auth 0 2024-03-25T18:20:34.508 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.507+0000 7f494affd640 10 monclient: _check_auth_tickets 2024-03-25T18:20:34.509 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.507+0000 7f494affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:34.509 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.507+0000 7f495c493640 5 monclient: authenticate success, global_id 4414 2024-03-25T18:20:34.509 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.507+0000 7f495c493640 10 monclient: _renew_subs 2024-03-25T18:20:34.509 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.507+0000 7f495c493640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:20:34.509 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.507+0000 7f495c493640 1 -- 172.21.15.97:0/3257994277 --> v1:172.21.15.98:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49541dab00 con 0x7f49540a98f0 2024-03-25T18:20:34.509 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.507+0000 7f495c493640 10 client.4414.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:20:34.509 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.507+0000 7f495c493640 10 monclient: _renew_subs 2024-03-25T18:20:34.509 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.507+0000 7f495c493640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:20:34.509 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.507+0000 7f495c493640 1 -- 172.21.15.97:0/3257994277 --> v1:172.21.15.98:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f495413b4f0 con 0x7f49540a98f0 2024-03-25T18:20:34.509 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.507+0000 7f495c493640 1 librados: init done 2024-03-25T18:20:34.510 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.508+0000 7f495c493640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:34.510 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.508+0000 7f495c493640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:34.510 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.508+0000 7f495c493640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:20:34.510 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.508+0000 7f495c493640 1 -- 172.21.15.97:0/3257994277 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49541dae00 con 0x7f49540a98f0 2024-03-25T18:20:34.528 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.528+0000 7f494affd640 1 -- 172.21.15.97:0/3257994277 <== mon.1 v1:172.21.15.98:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4950004e10 con 0x7f49540a98f0 2024-03-25T18:20:34.528 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.528+0000 7f494affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:34.528 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.528+0000 7f494affd640 1 -- 172.21.15.97:0/3257994277 <== mon.1 v1:172.21.15.98:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f49500051a0 con 0x7f49540a98f0 2024-03-25T18:20:34.528 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.528+0000 7f494affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:34.528 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.528+0000 7f494affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:20:34.528 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.528+0000 7f494affd640 10 monclient: dump: 2024-03-25T18:20:34.528 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:20:34.528 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:20:34.529 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:20:34.529 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:20:34.529 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:34.529 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:20:34.529 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:20:34.529 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:20:34.529 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:20:34.529 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:20:34.529 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.528+0000 7f4958a2c640 4 set_mon_vals no callback set 2024-03-25T18:20:34.530 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.528+0000 7f494affd640 1 -- 172.21.15.97:0/3257994277 <== mon.1 v1:172.21.15.98:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2377622137 0 0) 0x7f49500053c0 con 0x7f49540a98f0 2024-03-25T18:20:34.530 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.529+0000 7f4959a2e640 10 monclient: get_auth_request con 0x7f493806de80 auth_method 0 2024-03-25T18:20:34.530 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.529+0000 7f494affd640 10 client.4414.objecter ms_handle_connect 0x7f493806de80 2024-03-25T18:20:34.531 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.531+0000 7f494affd640 1 -- 172.21.15.97:0/3257994277 <== mon.1 v1:172.21.15.98:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 430711918 0 0) 0x7f4950054d40 con 0x7f49540a98f0 2024-03-25T18:20:34.531 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.531+0000 7f494affd640 10 client.4414.objecter ms_dispatch 0x7f4954000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:20:34.531 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.531+0000 7f494affd640 3 client.4414.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:20:34.531 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.531+0000 7f494affd640 3 client.4414.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:20:34.531 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.531+0000 7f494affd640 20 client.4414.objecter dump_active .. 0 homeless 2024-03-25T18:20:34.784 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.784+0000 7f494affd640 1 -- 172.21.15.97:0/3257994277 <== mon.1 v1:172.21.15.98:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4950050850 con 0x7f49540a98f0 2024-03-25T18:20:34.784 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.784+0000 7f494affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:34.784 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.784+0000 7f494affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:20:34.997 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.996+0000 7f495c493640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:34.997 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.996+0000 7f495c493640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:34.997 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.996+0000 7f495c493640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:20:34.997 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:34.996+0000 7f495c493640 1 -- 172.21.15.97:0/3257994277 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f49540b3e80 con 0x7f49540a98f0 2024-03-25T18:20:35.029 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:35.029+0000 7f494affd640 1 -- 172.21.15.97:0/3257994277 <== mon.1 v1:172.21.15.98:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2169427412) 0x7f4950052780 con 0x7f49540a98f0 2024-03-25T18:20:35.030 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:35.029+0000 7f494affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:35.030 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:35.029+0000 7f494affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:20:35.030 INFO:teuthology.misc.health.smithi097.stdout: 2024-03-25T18:20:35.030 INFO:teuthology.misc.health.smithi097.stdout:{"epoch":10,"fsid":"4c3d472c-79b8-486b-9e3d-2738d4306754","created":"2024-03-25T18:16:14.737309+0000","modified":"2024-03-25T18:16:20.357492+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:20.357492+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cb5152b9-262f-45c3-94b8-18d0e81df558","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6daf8793-0838-473d-a7d6-9a0d2894c75d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"136f1493-3b9c-4f43-a2e7-641482d69dae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"397d538c-4f64-4daf-9e8b-662d09f91327","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4f1df08-663f-4bda-93f8-3878ab7c24b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c4b18d70-a3ce-4b15-9b70-eae5525bd761","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8dd4033a-de0e-4f8a-96dc-59f3c1f825b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0077722e-ffa6-4a3d-9891-929d9869b286","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:20:35.032 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:35.031+0000 7f495c493640 10 librados: watch_flush enter 2024-03-25T18:20:35.032 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:35.031+0000 7f495c493640 10 librados: watch_flush exit 2024-03-25T18:20:35.032 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:35.031+0000 7f495c493640 20 client.4414.objecter shutdown clearing up homeless session... 2024-03-25T18:20:35.032 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:35.031+0000 7f495c493640 10 client.4414.objecter successfully canceled tick 2024-03-25T18:20:35.032 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:35.031+0000 7f495c493640 1 -- 172.21.15.97:0/3257994277 >> v1:172.21.15.98:6800/800387573 conn(0x7f493806de80 legacy=0x7f4938070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:35.032 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:35.031+0000 7f495c493640 10 monclient: shutdown 2024-03-25T18:20:35.032 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:35.031+0000 7f495c493640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:35.033 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:35.031+0000 7f495c493640 1 -- 172.21.15.97:0/3257994277 >> v1:172.21.15.98:6789/0 conn(0x7f49540a98f0 legacy=0x7f4954137370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:35.033 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:35.031+0000 7f495c493640 1 -- 172.21.15.97:0/3257994277 shutdown_connections 2024-03-25T18:20:35.033 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:35.031+0000 7f495c493640 1 -- 172.21.15.97:0/3257994277 >> 172.21.15.97:0/3257994277 conn(0x7f49540b7ea0 msgr2=0x7f4954074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:35.033 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:35.031+0000 7f495c493640 1 -- 172.21.15.97:0/3257994277 shutdown_connections 2024-03-25T18:20:35.033 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:35.032+0000 7f495c493640 1 -- 172.21.15.97:0/3257994277 wait complete. 2024-03-25T18:20:35.033 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:35.032+0000 7f495c493640 1 librados: shutdown 2024-03-25T18:20:35.044 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:20:41.849 DEBUG:teuthology.orchestra.run.smithi097:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:20:41.963 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:41.961+0000 7efe36663640 10 monclient: get_monmap_and_config 2024-03-25T18:20:41.963 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:41.961+0000 7efe36663640 10 monclient: build_initial_monmap 2024-03-25T18:20:41.964 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:41.961+0000 7efe36663640 10 monclient: monmap: 2024-03-25T18:20:41.964 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:20:41.964 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:20:41.964 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:20:41.963222+0000 2024-03-25T18:20:41.964 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:20:41.963222+0000 2024-03-25T18:20:41.964 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:41.964 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:20:41.964 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:20:41.964 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T18:20:41.964 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-b 2024-03-25T18:20:41.965 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:20:41.966 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:41.962+0000 7efe36663640 1 Processor -- start 2024-03-25T18:20:41.966 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:41.963+0000 7efe36663640 1 -- start start 2024-03-25T18:20:41.966 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:41.963+0000 7efe36663640 10 monclient: init 2024-03-25T18:20:41.966 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:41.963+0000 7efe36663640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:41.966 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:41.963+0000 7efe36663640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:20:41.966 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:41.963+0000 7efe36663640 10 monclient(hunting): picked mon.noname-a con 0x7efe300a98f0 addr v1:172.21.15.97:6789/0 2024-03-25T18:20:41.966 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:41.964+0000 7efe36663640 10 monclient(hunting): picked mon.noname-c con 0x7efe3013f710 addr v1:172.21.15.98:6789/0 2024-03-25T18:20:41.966 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:41.964+0000 7efe36663640 10 monclient(hunting): picked mon.noname-b con 0x7efe301430c0 addr v1:172.21.15.97:6790/0 2024-03-25T18:20:41.966 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:41.964+0000 7efe36663640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe30142520 con 0x7efe300a98f0 2024-03-25T18:20:41.967 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:41.964+0000 7efe36663640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe300b28a0 con 0x7efe3013f710 2024-03-25T18:20:41.967 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:41.964+0000 7efe36663640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe300b2a30 con 0x7efe301430c0 2024-03-25T18:20:41.967 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:41.964+0000 7efe36663640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:41.967 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:41.964+0000 7efe36663640 10 monclient(hunting): authenticate will time out at 7854.240017s 2024-03-25T18:20:41.976 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:41.975+0000 7efe2f7fe640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7efe3013f710 0x7efe301429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.97:56486/0 (socket says 172.21.15.97:56486) 2024-03-25T18:20:41.976 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:41.975+0000 7efe2f7fe640 1 -- 172.21.15.97:0/1330088835 learned_addr learned my addr 172.21.15.97:0/1330088835 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:20:41.992 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:41.991+0000 7efe2effd640 1 -- 172.21.15.97:0/1330088835 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2718662833 0 0) 0x7efe300b28a0 con 0x7efe3013f710 2024-03-25T18:20:41.992 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:41.991+0000 7efe2effd640 10 monclient(hunting): _negotiate 2024-03-25T18:20:41.992 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:41.991+0000 7efe2effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:41.993 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:41.991+0000 7efe2effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:41.993 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:41.991+0000 7efe2effd640 10 monclient(hunting): my global_id is 4417 2024-03-25T18:20:41.993 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:41.991+0000 7efe2effd640 1 -- 172.21.15.97:0/1330088835 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe18003520 con 0x7efe3013f710 2024-03-25T18:20:41.993 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:41.991+0000 7efe2effd640 1 -- 172.21.15.97:0/1330088835 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2617463532 0 0) 0x7efe300b2a30 con 0x7efe301430c0 2024-03-25T18:20:41.993 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:41.991+0000 7efe2effd640 10 monclient(hunting): _negotiate 2024-03-25T18:20:41.993 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:41.991+0000 7efe2effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:41.993 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:41.991+0000 7efe2effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:41.993 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:41.991+0000 7efe2effd640 10 monclient(hunting): my global_id is 4409 2024-03-25T18:20:41.993 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:41.991+0000 7efe2effd640 1 -- 172.21.15.97:0/1330088835 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe300b28a0 con 0x7efe301430c0 2024-03-25T18:20:42.002 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:42.001+0000 7efe2effd640 1 -- 172.21.15.97:0/1330088835 <== mon.2 v1:172.21.15.97:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 23665568 0 0) 0x7efe300b28a0 con 0x7efe301430c0 2024-03-25T18:20:42.003 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:42.001+0000 7efe2effd640 1 -- 172.21.15.97:0/1330088835 >> v1:172.21.15.98:6789/0 conn(0x7efe3013f710 legacy=0x7efe301429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:42.003 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:42.001+0000 7efe2effd640 1 -- 172.21.15.97:0/1330088835 >> v1:172.21.15.97:6789/0 conn(0x7efe300a98f0 legacy=0x7efe300a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:42.003 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:42.001+0000 7efe2effd640 10 monclient: _finish_hunting 0 2024-03-25T18:20:42.003 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:42.001+0000 7efe2effd640 1 monclient: found mon.noname-b 2024-03-25T18:20:42.003 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:42.001+0000 7efe2effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:42.003 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:42.001+0000 7efe2effd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.97:6790/0 2024-03-25T18:20:42.003 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:42.001+0000 7efe2effd640 1 -- 172.21.15.97:0/1330088835 --> v1:172.21.15.97:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe300b44c0 con 0x7efe301430c0 2024-03-25T18:20:42.003 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:42.001+0000 7efe2effd640 10 monclient: _finish_auth 0 2024-03-25T18:20:42.003 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:42.001+0000 7efe2effd640 10 monclient: _check_auth_tickets 2024-03-25T18:20:42.003 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:42.001+0000 7efe2effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:42.003 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:42.001+0000 7efe2effd640 1 -- 172.21.15.97:0/1330088835 <== mon.2 v1:172.21.15.97:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7efe280019f0 con 0x7efe301430c0 2024-03-25T18:20:42.003 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:42.001+0000 7efe2effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:42.003 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:42.001+0000 7efe2effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:20:42.003 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:42.001+0000 7efe2effd640 10 monclient: dump: 2024-03-25T18:20:42.004 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:20:42.004 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:20:42.004 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:20:42.004 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:20:42.004 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:42.004 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:20:42.004 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:20:42.004 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:20:42.004 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:20:42.004 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:20:42.004 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:42.001+0000 7efe36663640 5 monclient: authenticate success, global_id 4409 2024-03-25T18:20:42.004 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:42.001+0000 7efe36663640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:20:42.023 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:42.021+0000 7efe2effd640 1 -- 172.21.15.97:0/1330088835 <== mon.2 v1:172.21.15.97:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efe28001b50 con 0x7efe301430c0 2024-03-25T18:20:42.023 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:42.021+0000 7efe2effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:42.023 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:42.021+0000 7efe2effd640 1 -- 172.21.15.97:0/1330088835 <== mon.2 v1:172.21.15.97:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7efe28001f20 con 0x7efe301430c0 2024-03-25T18:20:42.023 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:42.021+0000 7efe36663640 10 monclient: get_monmap_and_config success 2024-03-25T18:20:42.023 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:42.021+0000 7efe36663640 4 set_mon_vals no callback set 2024-03-25T18:20:42.023 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:42.021+0000 7efe2effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:42.023 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:42.021+0000 7efe2effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:20:42.023 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:42.021+0000 7efe2effd640 10 monclient: dump: 2024-03-25T18:20:42.023 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:20:42.023 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:20:42.023 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:20:42.023 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:20:42.023 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:42.023 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:20:42.024 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:20:42.024 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:20:42.024 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:20:42.024 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:20:42.026 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:42.022+0000 7efe36663640 10 monclient: shutdown 2024-03-25T18:20:42.026 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:42.022+0000 7efe36663640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:42.026 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:42.022+0000 7efe36663640 1 -- 172.21.15.97:0/1330088835 >> v1:172.21.15.97:6790/0 conn(0x7efe301430c0 legacy=0x7efe3014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:42.026 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:42.022+0000 7efe36663640 1 -- 172.21.15.97:0/1330088835 shutdown_connections 2024-03-25T18:20:42.026 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:42.022+0000 7efe36663640 1 -- 172.21.15.97:0/1330088835 >> 172.21.15.97:0/1330088835 conn(0x7efe300b7ea0 msgr2=0x7efe300b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:42.026 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:42.022+0000 7efe36663640 1 -- 172.21.15.97:0/1330088835 shutdown_connections 2024-03-25T18:20:42.026 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:42.022+0000 7efe36663640 1 -- 172.21.15.97:0/1330088835 wait complete. 2024-03-25T18:20:42.026 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:42.023+0000 7efe36663640 10 monclient: build_initial_monmap 2024-03-25T18:20:42.027 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:42.023+0000 7efe36663640 10 monclient: monmap: 2024-03-25T18:20:42.027 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:20:42.027 INFO:teuthology.misc.health.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:20:42.027 INFO:teuthology.misc.health.smithi097.stderr:last_changed 0.000000 2024-03-25T18:20:42.027 INFO:teuthology.misc.health.smithi097.stderr:created 0.000000 2024-03-25T18:20:42.027 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:42.027 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 1 2024-03-25T18:20:42.027 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:20:42.027 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T18:20:42.027 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-c 2024-03-25T18:20:42.027 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:20:42.027 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:42.023+0000 7efe36663640 1 librados: starting msgr at 2024-03-25T18:20:42.027 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:42.023+0000 7efe36663640 1 librados: starting objecter 2024-03-25T18:20:42.027 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:42.023+0000 7efe36663640 1 Processor -- start 2024-03-25T18:20:42.027 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:42.024+0000 7efe36663640 1 -- start start 2024-03-25T18:20:42.028 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:42.024+0000 7efe36663640 1 librados: setting wanted keys 2024-03-25T18:20:42.028 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:42.024+0000 7efe36663640 1 librados: calling monclient init 2024-03-25T18:20:42.028 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:42.024+0000 7efe36663640 10 monclient: init 2024-03-25T18:20:42.028 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:42.024+0000 7efe36663640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:42.028 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:42.024+0000 7efe36663640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:20:42.028 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:42.024+0000 7efe36663640 10 monclient(hunting): picked mon.noname-b con 0x7efe300a98f0 addr v1:172.21.15.98:6789/0 2024-03-25T18:20:42.028 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:42.024+0000 7efe36663640 10 monclient(hunting): picked mon.noname-a con 0x7efe3013f710 addr v1:172.21.15.97:6789/0 2024-03-25T18:20:42.028 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:42.024+0000 7efe36663640 10 monclient(hunting): picked mon.noname-c con 0x7efe301430c0 addr v1:172.21.15.97:6790/0 2024-03-25T18:20:42.028 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:42.024+0000 7efe36663640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe30071bc0 con 0x7efe3013f710 2024-03-25T18:20:42.028 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:42.024+0000 7efe36663640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe300720a0 con 0x7efe300a98f0 2024-03-25T18:20:42.028 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:42.024+0000 7efe36663640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe3006f4e0 con 0x7efe301430c0 2024-03-25T18:20:42.028 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:42.024+0000 7efe36663640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:42.028 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:42.024+0000 7efe36663640 10 monclient(hunting): authenticate will time out at 7854.300803s 2024-03-25T18:20:42.030 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:42.029+0000 7efe2ffff640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7efe300a98f0 0x7efe3006edd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.97:56522/0 (socket says 172.21.15.97:56522) 2024-03-25T18:20:42.030 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:42.029+0000 7efe2ffff640 1 -- 172.21.15.97:0/396338414 learned_addr learned my addr 172.21.15.97:0/396338414 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:20:42.031 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:42.030+0000 7efe2cff9640 10 client.?.objecter ms_handle_connect 0x7efe300a98f0 2024-03-25T18:20:42.031 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:42.030+0000 7efe2cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:42.041 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:42.040+0000 7efe2cff9640 1 -- 172.21.15.97:0/396338414 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1789245072 0 0) 0x7efe300720a0 con 0x7efe300a98f0 2024-03-25T18:20:42.041 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:42.040+0000 7efe2cff9640 10 monclient(hunting): _negotiate 2024-03-25T18:20:42.041 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:42.040+0000 7efe2cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:42.041 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:42.040+0000 7efe2cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:42.041 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:42.040+0000 7efe2cff9640 10 monclient(hunting): my global_id is 4420 2024-03-25T18:20:42.041 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:42.040+0000 7efe2cff9640 1 -- 172.21.15.97:0/396338414 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe1c004eb0 con 0x7efe300a98f0 2024-03-25T18:20:42.048 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:42.046+0000 7efe2cff9640 10 client.?.objecter ms_handle_connect 0x7efe301430c0 2024-03-25T18:20:42.048 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:42.046+0000 7efe2cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:42.048 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:42.047+0000 7efe2cff9640 1 -- 172.21.15.97:0/396338414 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2091026949 0 0) 0x7efe1c004eb0 con 0x7efe300a98f0 2024-03-25T18:20:42.049 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:42.047+0000 7efe2cff9640 1 -- 172.21.15.97:0/396338414 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efe300720a0 con 0x7efe300a98f0 2024-03-25T18:20:42.049 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:42.047+0000 7efe2cff9640 1 -- 172.21.15.97:0/396338414 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7efe20001a30 con 0x7efe300a98f0 2024-03-25T18:20:42.049 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:42.047+0000 7efe2cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:42.049 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:42.047+0000 7efe2cff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:20:42.049 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:42.047+0000 7efe2cff9640 10 monclient(hunting): dump: 2024-03-25T18:20:42.049 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:20:42.049 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:20:42.049 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:20:42.049 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:20:42.049 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:42.049 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:20:42.049 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:20:42.049 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:20:42.049 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:20:42.050 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:20:42.050 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:42.047+0000 7efe2cff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:20:42.050 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:42.049+0000 7efe2cff9640 10 client.?.objecter ms_handle_connect 0x7efe3013f710 2024-03-25T18:20:42.050 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:42.049+0000 7efe2cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:42.066 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:42.064+0000 7efe2cff9640 1 -- 172.21.15.97:0/396338414 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1122197466 0 0) 0x7efe30071bc0 con 0x7efe3013f710 2024-03-25T18:20:42.066 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:42.064+0000 7efe2cff9640 10 monclient(hunting): _negotiate 2024-03-25T18:20:42.066 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:42.064+0000 7efe2cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:42.066 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:42.064+0000 7efe2cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:42.066 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:42.064+0000 7efe2cff9640 10 monclient(hunting): my global_id is 4422 2024-03-25T18:20:42.066 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:42.064+0000 7efe2cff9640 1 -- 172.21.15.97:0/396338414 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe1c004eb0 con 0x7efe3013f710 2024-03-25T18:20:42.066 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:42.064+0000 7efe2cff9640 1 -- 172.21.15.97:0/396338414 <== mon.1 v1:172.21.15.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2508540774 0 0) 0x7efe300720a0 con 0x7efe300a98f0 2024-03-25T18:20:42.066 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:42.064+0000 7efe2cff9640 1 -- 172.21.15.97:0/396338414 >> v1:172.21.15.97:6790/0 conn(0x7efe301430c0 legacy=0x7efe30070b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:42.066 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:42.064+0000 7efe2cff9640 1 -- 172.21.15.97:0/396338414 >> v1:172.21.15.97:6789/0 conn(0x7efe3013f710 legacy=0x7efe30138f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:42.067 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:42.064+0000 7efe2cff9640 10 monclient: _finish_hunting 0 2024-03-25T18:20:42.067 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:42.064+0000 7efe2cff9640 1 monclient: found mon.b 2024-03-25T18:20:42.067 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:42.064+0000 7efe2cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:42.067 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:42.064+0000 7efe2cff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:20:42.067 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:42.064+0000 7efe2cff9640 1 -- 172.21.15.97:0/396338414 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe3006f670 con 0x7efe300a98f0 2024-03-25T18:20:42.067 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:42.064+0000 7efe2cff9640 10 monclient: _finish_auth 0 2024-03-25T18:20:42.067 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:42.064+0000 7efe2cff9640 10 monclient: _check_auth_tickets 2024-03-25T18:20:42.067 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:42.064+0000 7efe2cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:42.067 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:42.064+0000 7efe36663640 5 monclient: authenticate success, global_id 4420 2024-03-25T18:20:42.067 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:42.064+0000 7efe36663640 10 monclient: _renew_subs 2024-03-25T18:20:42.067 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:42.064+0000 7efe36663640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:20:42.067 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:42.064+0000 7efe36663640 1 -- 172.21.15.97:0/396338414 --> v1:172.21.15.98:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe30139670 con 0x7efe300a98f0 2024-03-25T18:20:42.067 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:42.064+0000 7efe36663640 10 client.4420.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:20:42.067 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:42.064+0000 7efe36663640 10 monclient: _renew_subs 2024-03-25T18:20:42.067 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:42.064+0000 7efe36663640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:20:42.068 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:42.064+0000 7efe36663640 1 -- 172.21.15.97:0/396338414 --> v1:172.21.15.98:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efe3006e030 con 0x7efe300a98f0 2024-03-25T18:20:42.068 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:42.064+0000 7efe36663640 1 librados: init done 2024-03-25T18:20:42.068 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:42.065+0000 7efe36663640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:42.068 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:42.065+0000 7efe36663640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:42.068 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:42.065+0000 7efe36663640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:20:42.068 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:42.065+0000 7efe36663640 1 -- 172.21.15.97:0/396338414 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe300704e0 con 0x7efe300a98f0 2024-03-25T18:20:42.343 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:42.341+0000 7efe2cff9640 1 -- 172.21.15.97:0/396338414 <== mon.1 v1:172.21.15.98:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efe20001d70 con 0x7efe300a98f0 2024-03-25T18:20:42.343 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:42.341+0000 7efe2cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:42.343 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:42.341+0000 7efe2cff9640 1 -- 172.21.15.97:0/396338414 <== mon.1 v1:172.21.15.98:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7efe20003a60 con 0x7efe300a98f0 2024-03-25T18:20:42.343 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:42.341+0000 7efe2cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:42.343 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:42.341+0000 7efe2cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:20:42.343 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:42.341+0000 7efe2cff9640 10 monclient: dump: 2024-03-25T18:20:42.343 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:20:42.343 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:20:42.343 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:20:42.343 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:20:42.343 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:42.344 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:20:42.344 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:20:42.344 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:20:42.344 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:20:42.344 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:20:42.344 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:42.341+0000 7efe2e7fc640 4 set_mon_vals no callback set 2024-03-25T18:20:42.344 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:42.342+0000 7efe2cff9640 1 -- 172.21.15.97:0/396338414 <== mon.1 v1:172.21.15.98:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2377622137 0 0) 0x7efe20003c80 con 0x7efe300a98f0 2024-03-25T18:20:42.345 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:42.343+0000 7efe2f7fe640 10 monclient: get_auth_request con 0x7efe1c06dd40 auth_method 0 2024-03-25T18:20:42.345 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:42.343+0000 7efe2cff9640 10 client.4420.objecter ms_handle_connect 0x7efe1c06dd40 2024-03-25T18:20:42.348 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:42.347+0000 7efe2cff9640 1 -- 172.21.15.97:0/396338414 <== mon.1 v1:172.21.15.98:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 430711918 0 0) 0x7efe200549d0 con 0x7efe300a98f0 2024-03-25T18:20:42.348 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:42.347+0000 7efe2cff9640 10 client.4420.objecter ms_dispatch 0x7efe30000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:20:42.348 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:42.347+0000 7efe2cff9640 3 client.4420.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:20:42.348 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:42.347+0000 7efe2cff9640 3 client.4420.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:20:42.348 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:42.347+0000 7efe2cff9640 20 client.4420.objecter dump_active .. 0 homeless 2024-03-25T18:20:42.349 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:42.347+0000 7efe2cff9640 1 -- 172.21.15.97:0/396338414 <== mon.1 v1:172.21.15.98:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7efe20050310 con 0x7efe300a98f0 2024-03-25T18:20:42.349 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:42.347+0000 7efe2cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:42.349 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:42.347+0000 7efe2cff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:20:42.601 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:42.599+0000 7efe36663640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:42.601 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:42.599+0000 7efe36663640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:42.601 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:42.599+0000 7efe36663640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:20:42.601 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:42.599+0000 7efe36663640 1 -- 172.21.15.97:0/396338414 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efe300627b0 con 0x7efe300a98f0 2024-03-25T18:20:42.628 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:42.626+0000 7efe2cff9640 1 -- 172.21.15.97:0/396338414 <== mon.1 v1:172.21.15.98:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2169427412) 0x7efe20052410 con 0x7efe300a98f0 2024-03-25T18:20:42.628 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:42.626+0000 7efe2cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:42.628 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:42.626+0000 7efe2cff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:20:42.628 INFO:teuthology.misc.health.smithi097.stdout: 2024-03-25T18:20:42.628 INFO:teuthology.misc.health.smithi097.stdout:{"epoch":10,"fsid":"4c3d472c-79b8-486b-9e3d-2738d4306754","created":"2024-03-25T18:16:14.737309+0000","modified":"2024-03-25T18:16:20.357492+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:20.357492+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cb5152b9-262f-45c3-94b8-18d0e81df558","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6daf8793-0838-473d-a7d6-9a0d2894c75d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"136f1493-3b9c-4f43-a2e7-641482d69dae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"397d538c-4f64-4daf-9e8b-662d09f91327","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4f1df08-663f-4bda-93f8-3878ab7c24b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c4b18d70-a3ce-4b15-9b70-eae5525bd761","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8dd4033a-de0e-4f8a-96dc-59f3c1f825b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0077722e-ffa6-4a3d-9891-929d9869b286","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:20:42.631 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:42.628+0000 7efe36663640 10 librados: watch_flush enter 2024-03-25T18:20:42.631 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:42.629+0000 7efe36663640 10 librados: watch_flush exit 2024-03-25T18:20:42.631 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:42.629+0000 7efe36663640 20 client.4420.objecter shutdown clearing up homeless session... 2024-03-25T18:20:42.631 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:42.629+0000 7efe36663640 10 client.4420.objecter successfully canceled tick 2024-03-25T18:20:42.631 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:42.629+0000 7efe36663640 1 -- 172.21.15.97:0/396338414 >> v1:172.21.15.98:6800/800387573 conn(0x7efe1c06dd40 legacy=0x7efe1c070200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:42.631 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:42.629+0000 7efe36663640 10 monclient: shutdown 2024-03-25T18:20:42.631 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:42.629+0000 7efe36663640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:42.631 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:42.629+0000 7efe36663640 1 -- 172.21.15.97:0/396338414 >> v1:172.21.15.98:6789/0 conn(0x7efe300a98f0 legacy=0x7efe3006edd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:42.631 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:42.629+0000 7efe36663640 1 -- 172.21.15.97:0/396338414 shutdown_connections 2024-03-25T18:20:42.631 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:42.629+0000 7efe36663640 1 -- 172.21.15.97:0/396338414 >> 172.21.15.97:0/396338414 conn(0x7efe300b7ea0 msgr2=0x7efe30074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:42.631 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:42.629+0000 7efe36663640 1 -- 172.21.15.97:0/396338414 shutdown_connections 2024-03-25T18:20:42.631 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:42.629+0000 7efe36663640 1 -- 172.21.15.97:0/396338414 wait complete. 2024-03-25T18:20:42.632 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:42.629+0000 7efe36663640 1 librados: shutdown 2024-03-25T18:20:42.643 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:20:49.450 DEBUG:teuthology.orchestra.run.smithi097:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:20:49.565 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.563+0000 7f910bd64640 10 monclient: get_monmap_and_config 2024-03-25T18:20:49.565 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.563+0000 7f910bd64640 10 monclient: build_initial_monmap 2024-03-25T18:20:49.565 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.563+0000 7f910bd64640 10 monclient: monmap: 2024-03-25T18:20:49.565 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:20:49.565 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:20:49.565 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:20:49.565170+0000 2024-03-25T18:20:49.565 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:20:49.565170+0000 2024-03-25T18:20:49.566 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:49.566 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:20:49.566 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:20:49.566 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T18:20:49.566 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-b 2024-03-25T18:20:49.566 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:20:49.567 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.564+0000 7f910bd64640 1 Processor -- start 2024-03-25T18:20:49.567 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.564+0000 7f910bd64640 1 -- start start 2024-03-25T18:20:49.568 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.564+0000 7f910bd64640 10 monclient: init 2024-03-25T18:20:49.568 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.565+0000 7f910bd64640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:49.568 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.565+0000 7f910bd64640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:20:49.568 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.565+0000 7f910bd64640 10 monclient(hunting): picked mon.noname-c con 0x7f91040b4d60 addr v1:172.21.15.98:6789/0 2024-03-25T18:20:49.568 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.565+0000 7f910bd64640 10 monclient(hunting): picked mon.noname-a con 0x7f91040b1730 addr v1:172.21.15.97:6789/0 2024-03-25T18:20:49.568 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.565+0000 7f910bd64640 10 monclient(hunting): picked mon.noname-b con 0x7f910414be90 addr v1:172.21.15.97:6790/0 2024-03-25T18:20:49.568 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.565+0000 7f910bd64640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91040af940 con 0x7f91040b1730 2024-03-25T18:20:49.568 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.565+0000 7f910bd64640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91040b0a70 con 0x7f91040b4d60 2024-03-25T18:20:49.568 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.565+0000 7f910bd64640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91040b0d50 con 0x7f910414be90 2024-03-25T18:20:49.568 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.565+0000 7f910bd64640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:49.568 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.565+0000 7f910bd64640 10 monclient(hunting): authenticate will time out at 7861.841585s 2024-03-25T18:20:49.582 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.581+0000 7f91092ff640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f91040b1730 0x7f91040b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:55918/0 (socket says 172.21.15.97:55918) 2024-03-25T18:20:49.582 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.581+0000 7f91092ff640 1 -- 172.21.15.97:0/3721298041 learned_addr learned my addr 172.21.15.97:0/3721298041 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:20:49.597 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.596+0000 7f9108afe640 1 -- 172.21.15.97:0/3721298041 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2933657614 0 0) 0x7f91040b0a70 con 0x7f91040b4d60 2024-03-25T18:20:49.597 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.596+0000 7f9108afe640 10 monclient(hunting): _negotiate 2024-03-25T18:20:49.597 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.596+0000 7f9108afe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:49.597 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.596+0000 7f9108afe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:49.597 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.596+0000 7f9108afe640 10 monclient(hunting): my global_id is 4423 2024-03-25T18:20:49.597 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.596+0000 7f9108afe640 1 -- 172.21.15.97:0/3721298041 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90f0003520 con 0x7f91040b4d60 2024-03-25T18:20:49.612 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.610+0000 7f9108afe640 1 -- 172.21.15.97:0/3721298041 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2100860460 0 0) 0x7f91040b0d50 con 0x7f910414be90 2024-03-25T18:20:49.612 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.610+0000 7f9108afe640 10 monclient(hunting): _negotiate 2024-03-25T18:20:49.612 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.610+0000 7f9108afe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:49.612 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.610+0000 7f9108afe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:49.612 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.610+0000 7f9108afe640 10 monclient(hunting): my global_id is 4415 2024-03-25T18:20:49.612 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.610+0000 7f9108afe640 1 -- 172.21.15.97:0/3721298041 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91040b0a70 con 0x7f910414be90 2024-03-25T18:20:49.617 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.615+0000 7f9108afe640 1 -- 172.21.15.97:0/3721298041 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1108592900 0 0) 0x7f91040af940 con 0x7f91040b1730 2024-03-25T18:20:49.617 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.615+0000 7f9108afe640 10 monclient(hunting): _negotiate 2024-03-25T18:20:49.617 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.615+0000 7f9108afe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:49.617 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.615+0000 7f9108afe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:49.617 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.615+0000 7f9108afe640 10 monclient(hunting): my global_id is 4425 2024-03-25T18:20:49.617 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.615+0000 7f9108afe640 1 -- 172.21.15.97:0/3721298041 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91040b0d50 con 0x7f91040b1730 2024-03-25T18:20:49.617 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.615+0000 7f9108afe640 1 -- 172.21.15.97:0/3721298041 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1199724564 0 0) 0x7f90f0003520 con 0x7f91040b4d60 2024-03-25T18:20:49.618 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.615+0000 7f9108afe640 1 -- 172.21.15.97:0/3721298041 >> v1:172.21.15.97:6790/0 conn(0x7f910414be90 legacy=0x7f910414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:49.618 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.615+0000 7f9108afe640 1 -- 172.21.15.97:0/3721298041 >> v1:172.21.15.97:6789/0 conn(0x7f91040b1730 legacy=0x7f91040b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:49.618 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.615+0000 7f9108afe640 10 monclient: _finish_hunting 0 2024-03-25T18:20:49.618 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.615+0000 7f9108afe640 1 monclient: found mon.noname-c 2024-03-25T18:20:49.618 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.615+0000 7f9108afe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:49.618 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.615+0000 7f9108afe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.98:6789/0 2024-03-25T18:20:49.618 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.615+0000 7f9108afe640 1 -- 172.21.15.97:0/3721298041 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91040b1030 con 0x7f91040b4d60 2024-03-25T18:20:49.618 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.615+0000 7f9108afe640 10 monclient: _finish_auth 0 2024-03-25T18:20:49.618 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.615+0000 7f9108afe640 10 monclient: _check_auth_tickets 2024-03-25T18:20:49.618 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.615+0000 7f9108afe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:49.618 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.615+0000 7f9108afe640 1 -- 172.21.15.97:0/3721298041 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f9100001690 con 0x7f91040b4d60 2024-03-25T18:20:49.618 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.615+0000 7f9108afe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:49.618 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.615+0000 7f9108afe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:20:49.619 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.615+0000 7f9108afe640 10 monclient: dump: 2024-03-25T18:20:49.619 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:20:49.619 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:20:49.619 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:20:49.619 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:20:49.619 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:49.619 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:20:49.619 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:20:49.619 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:20:49.619 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:20:49.619 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:20:49.619 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.615+0000 7f910bd64640 5 monclient: authenticate success, global_id 4423 2024-03-25T18:20:49.619 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.615+0000 7f910bd64640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:20:49.637 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.636+0000 7f9108afe640 1 -- 172.21.15.97:0/3721298041 <== mon.1 v1:172.21.15.98:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f91000017f0 con 0x7f91040b4d60 2024-03-25T18:20:49.637 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.636+0000 7f9108afe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:49.638 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.636+0000 7f9108afe640 1 -- 172.21.15.97:0/3721298041 <== mon.1 v1:172.21.15.98:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f9100001c00 con 0x7f91040b4d60 2024-03-25T18:20:49.638 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.636+0000 7f910bd64640 10 monclient: get_monmap_and_config success 2024-03-25T18:20:49.638 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.636+0000 7f910bd64640 4 set_mon_vals no callback set 2024-03-25T18:20:49.638 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.636+0000 7f9108afe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:49.638 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.636+0000 7f9108afe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:20:49.638 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.636+0000 7f9108afe640 10 monclient: dump: 2024-03-25T18:20:49.638 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:20:49.638 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:20:49.638 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:20:49.638 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:20:49.638 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:49.639 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:20:49.639 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:20:49.639 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:20:49.639 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:20:49.639 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:20:49.641 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.637+0000 7f910bd64640 10 monclient: shutdown 2024-03-25T18:20:49.641 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.637+0000 7f910bd64640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:49.641 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.637+0000 7f910bd64640 1 -- 172.21.15.97:0/3721298041 >> v1:172.21.15.98:6789/0 conn(0x7f91040b4d60 legacy=0x7f910413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:49.641 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.637+0000 7f910bd64640 1 -- 172.21.15.97:0/3721298041 shutdown_connections 2024-03-25T18:20:49.641 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.637+0000 7f910bd64640 1 -- 172.21.15.97:0/3721298041 >> 172.21.15.97:0/3721298041 conn(0x7f91040617a0 msgr2=0x7f9104061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:49.641 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.637+0000 7f910bd64640 1 -- 172.21.15.97:0/3721298041 shutdown_connections 2024-03-25T18:20:49.641 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.637+0000 7f910bd64640 1 -- 172.21.15.97:0/3721298041 wait complete. 2024-03-25T18:20:49.642 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.638+0000 7f910bd64640 10 monclient: build_initial_monmap 2024-03-25T18:20:49.642 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.638+0000 7f910bd64640 10 monclient: monmap: 2024-03-25T18:20:49.642 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:20:49.642 INFO:teuthology.misc.health.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:20:49.642 INFO:teuthology.misc.health.smithi097.stderr:last_changed 0.000000 2024-03-25T18:20:49.642 INFO:teuthology.misc.health.smithi097.stderr:created 0.000000 2024-03-25T18:20:49.642 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:49.642 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 1 2024-03-25T18:20:49.642 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:20:49.642 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T18:20:49.642 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-c 2024-03-25T18:20:49.642 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:20:49.642 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.638+0000 7f910bd64640 1 librados: starting msgr at 2024-03-25T18:20:49.643 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.638+0000 7f910bd64640 1 librados: starting objecter 2024-03-25T18:20:49.643 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.638+0000 7f910bd64640 1 Processor -- start 2024-03-25T18:20:49.643 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.638+0000 7f910bd64640 1 -- start start 2024-03-25T18:20:49.643 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.638+0000 7f910bd64640 1 librados: setting wanted keys 2024-03-25T18:20:49.643 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.638+0000 7f910bd64640 1 librados: calling monclient init 2024-03-25T18:20:49.643 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.638+0000 7f910bd64640 10 monclient: init 2024-03-25T18:20:49.643 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.639+0000 7f910bd64640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:49.643 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.639+0000 7f910bd64640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:20:49.643 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.639+0000 7f910bd64640 10 monclient(hunting): picked mon.noname-b con 0x7f91040b1730 addr v1:172.21.15.98:6789/0 2024-03-25T18:20:49.643 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.639+0000 7f910bd64640 10 monclient(hunting): picked mon.noname-c con 0x7f91040b4d60 addr v1:172.21.15.97:6790/0 2024-03-25T18:20:49.643 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.639+0000 7f910bd64640 10 monclient(hunting): picked mon.noname-a con 0x7f910414be90 addr v1:172.21.15.97:6789/0 2024-03-25T18:20:49.643 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.639+0000 7f910bd64640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91041d8750 con 0x7f910414be90 2024-03-25T18:20:49.643 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.639+0000 7f910bd64640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9104140a90 con 0x7f91040b1730 2024-03-25T18:20:49.643 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.639+0000 7f910bd64640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9104140c20 con 0x7f91040b4d60 2024-03-25T18:20:49.643 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.639+0000 7f910bd64640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:49.644 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.639+0000 7f910bd64640 10 monclient(hunting): authenticate will time out at 7861.915465s 2024-03-25T18:20:49.647 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.646+0000 7f910a301640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f910414be90 0x7f91041e21a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:55976/0 (socket says 172.21.15.97:55976) 2024-03-25T18:20:49.648 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.646+0000 7f910a301640 1 -- 172.21.15.97:0/351899156 learned_addr learned my addr 172.21.15.97:0/351899156 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:20:49.649 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.647+0000 7f90fa7fc640 10 client.?.objecter ms_handle_connect 0x7f910414be90 2024-03-25T18:20:49.649 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.647+0000 7f90fa7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:49.649 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.648+0000 7f90fa7fc640 10 client.?.objecter ms_handle_connect 0x7f91040b4d60 2024-03-25T18:20:49.650 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.648+0000 7f90fa7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:49.656 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.655+0000 7f90fa7fc640 10 client.?.objecter ms_handle_connect 0x7f91040b1730 2024-03-25T18:20:49.656 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.655+0000 7f90fa7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:49.658 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.657+0000 7f90fa7fc640 1 -- 172.21.15.97:0/351899156 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1946470901 0 0) 0x7f91041d8750 con 0x7f910414be90 2024-03-25T18:20:49.658 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.657+0000 7f90fa7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:20:49.658 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.657+0000 7f90fa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:49.658 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.657+0000 7f90fa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:49.658 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.657+0000 7f90fa7fc640 10 monclient(hunting): my global_id is 4428 2024-03-25T18:20:49.658 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.657+0000 7f90fa7fc640 1 -- 172.21.15.97:0/351899156 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90e8004eb0 con 0x7f910414be90 2024-03-25T18:20:49.659 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.658+0000 7f90fa7fc640 1 -- 172.21.15.97:0/351899156 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1455378411 0 0) 0x7f9104140a90 con 0x7f91040b1730 2024-03-25T18:20:49.659 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.658+0000 7f90fa7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:20:49.659 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.658+0000 7f90fa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:49.659 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.658+0000 7f90fa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:49.660 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.658+0000 7f90fa7fc640 10 monclient(hunting): my global_id is 4426 2024-03-25T18:20:49.660 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.658+0000 7f90fa7fc640 1 -- 172.21.15.97:0/351899156 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91041d8750 con 0x7f91040b1730 2024-03-25T18:20:49.662 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.661+0000 7f90fa7fc640 1 -- 172.21.15.97:0/351899156 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1714615796 0 0) 0x7f9104140c20 con 0x7f91040b4d60 2024-03-25T18:20:49.662 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.661+0000 7f90fa7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:20:49.662 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.661+0000 7f90fa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:49.662 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.661+0000 7f90fa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:49.662 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.661+0000 7f90fa7fc640 10 monclient(hunting): my global_id is 4418 2024-03-25T18:20:49.662 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.661+0000 7f90fa7fc640 1 -- 172.21.15.97:0/351899156 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9104140a90 con 0x7f91040b4d60 2024-03-25T18:20:49.670 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.668+0000 7f90fa7fc640 1 -- 172.21.15.97:0/351899156 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3786255745 0 0) 0x7f91041d8750 con 0x7f91040b1730 2024-03-25T18:20:49.670 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.668+0000 7f90fa7fc640 1 -- 172.21.15.97:0/351899156 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9104140c20 con 0x7f91040b1730 2024-03-25T18:20:49.670 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.668+0000 7f90fa7fc640 1 -- 172.21.15.97:0/351899156 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f9100001b00 con 0x7f91040b1730 2024-03-25T18:20:49.670 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.668+0000 7f90fa7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:49.670 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.668+0000 7f90fa7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:20:49.670 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.668+0000 7f90fa7fc640 10 monclient(hunting): dump: 2024-03-25T18:20:49.670 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:20:49.670 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:20:49.670 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:20:49.670 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:20:49.670 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:49.671 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:20:49.671 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:20:49.671 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:20:49.671 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:20:49.671 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:20:49.671 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.668+0000 7f90fa7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:20:49.681 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.679+0000 7f90fa7fc640 1 -- 172.21.15.97:0/351899156 <== mon.1 v1:172.21.15.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2922596291 0 0) 0x7f9104140c20 con 0x7f91040b1730 2024-03-25T18:20:49.681 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.679+0000 7f90fa7fc640 1 -- 172.21.15.97:0/351899156 >> v1:172.21.15.97:6790/0 conn(0x7f91040b4d60 legacy=0x7f9104140380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:49.681 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.679+0000 7f90fa7fc640 1 -- 172.21.15.97:0/351899156 >> v1:172.21.15.97:6789/0 conn(0x7f910414be90 legacy=0x7f91041e21a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:49.681 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.679+0000 7f90fa7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:20:49.681 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.679+0000 7f90fa7fc640 1 monclient: found mon.b 2024-03-25T18:20:49.681 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.679+0000 7f90fa7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:49.681 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.679+0000 7f90fa7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:20:49.681 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.679+0000 7f90fa7fc640 1 -- 172.21.15.97:0/351899156 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f910413b900 con 0x7f91040b1730 2024-03-25T18:20:49.681 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.679+0000 7f90fa7fc640 10 monclient: _finish_auth 0 2024-03-25T18:20:49.681 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.679+0000 7f90fa7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:20:49.682 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.679+0000 7f90fa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:49.682 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.679+0000 7f910bd64640 5 monclient: authenticate success, global_id 4426 2024-03-25T18:20:49.682 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.679+0000 7f910bd64640 10 monclient: _renew_subs 2024-03-25T18:20:49.682 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.679+0000 7f910bd64640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:20:49.682 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.679+0000 7f910bd64640 1 -- 172.21.15.97:0/351899156 --> v1:172.21.15.98:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f910413bb70 con 0x7f91040b1730 2024-03-25T18:20:49.682 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.679+0000 7f910bd64640 10 client.4426.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:20:49.682 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.679+0000 7f910bd64640 10 monclient: _renew_subs 2024-03-25T18:20:49.682 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.679+0000 7f910bd64640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:20:49.682 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.679+0000 7f910bd64640 1 -- 172.21.15.97:0/351899156 --> v1:172.21.15.98:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f91041e3aa0 con 0x7f91040b1730 2024-03-25T18:20:49.682 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.679+0000 7f910bd64640 1 librados: init done 2024-03-25T18:20:49.683 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.680+0000 7f910bd64640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:49.683 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.680+0000 7f910bd64640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:49.683 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.680+0000 7f910bd64640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:20:49.683 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.680+0000 7f910bd64640 1 -- 172.21.15.97:0/351899156 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91041d7e90 con 0x7f91040b1730 2024-03-25T18:20:49.687 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.685+0000 7f9109b00640 1 -- 172.21.15.97:0/351899156 >> v1:172.21.15.98:6789/0 conn(0x7f91040b1730 legacy=0x7f91041d8040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-25T18:20:49.688 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.685+0000 7f9109b00640 1 -- 172.21.15.97:0/351899156 >> v1:172.21.15.98:6789/0 conn(0x7f91040b1730 legacy=0x7f91041d8040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T18:20:49.688 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.685+0000 7f9109b00640 1 --1- 172.21.15.97:0/351899156 >> v1:172.21.15.98:6789/0 conn(0x7f91040b1730 0x7f91041d8040 :-1 s=OPENED pgs=31470 cs=1 l=1).handle_message read tag failed 2024-03-25T18:20:49.688 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.685+0000 7f9109b00640 1 --1- 172.21.15.97:0/351899156 >> v1:172.21.15.98:6789/0 conn(0x7f91040b1730 0x7f91041d8040 :-1 s=OPENED pgs=31470 cs=1 l=1).fault on lossy channel, failing 2024-03-25T18:20:49.688 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.685+0000 7f90fa7fc640 10 monclient: ms_handle_reset current mon v1:172.21.15.98:6789/0 2024-03-25T18:20:49.688 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.685+0000 7f90fa7fc640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:49.688 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.685+0000 7f90fa7fc640 1 -- 172.21.15.97:0/351899156 >> v1:172.21.15.98:6789/0 conn(0x7f91040b1730 legacy=0x7f91041d8040 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T18:20:49.688 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.685+0000 7f90fa7fc640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:20:49.688 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.685+0000 7f90fa7fc640 10 monclient(hunting): picked mon.a con 0x7f910414be90 addr v1:172.21.15.97:6789/0 2024-03-25T18:20:49.688 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.685+0000 7f90fa7fc640 10 monclient(hunting): picked mon.c con 0x7f91040b4d60 addr v1:172.21.15.97:6790/0 2024-03-25T18:20:49.688 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.685+0000 7f90fa7fc640 10 monclient(hunting): picked mon.b con 0x7f90e800da30 addr v1:172.21.15.98:6789/0 2024-03-25T18:20:49.688 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.685+0000 7f90fa7fc640 1 -- 172.21.15.97:0/351899156 --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f9104140c20 con 0x7f910414be90 2024-03-25T18:20:49.688 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.686+0000 7f90fa7fc640 1 -- 172.21.15.97:0/351899156 --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f91041d8750 con 0x7f90e800da30 2024-03-25T18:20:49.688 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.686+0000 7f90fa7fc640 1 -- 172.21.15.97:0/351899156 --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f90e8013a00 con 0x7f91040b4d60 2024-03-25T18:20:49.688 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.686+0000 7f90fa7fc640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:49.700 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.699+0000 7f90fa7fc640 10 client.4426.objecter ms_handle_connect 0x7f910414be90 2024-03-25T18:20:49.700 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.699+0000 7f90fa7fc640 10 client.4426.objecter resend_mon_ops 2024-03-25T18:20:49.706 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.705+0000 7f90fa7fc640 1 -- 172.21.15.97:0/351899156 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1940380901 0 0) 0x7f9104140c20 con 0x7f910414be90 2024-03-25T18:20:49.706 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.705+0000 7f90fa7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:20:49.706 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.705+0000 7f90fa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:49.706 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.705+0000 7f90fa7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-25T18:20:49.706 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.705+0000 7f90fa7fc640 1 -- 172.21.15.97:0/351899156 --> v1:172.21.15.97:6789/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f90e8011900 con 0x7f910414be90 2024-03-25T18:20:49.708 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.707+0000 7f90fa7fc640 10 client.4426.objecter ms_handle_connect 0x7f91040b4d60 2024-03-25T18:20:49.709 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.707+0000 7f90fa7fc640 10 client.4426.objecter resend_mon_ops 2024-03-25T18:20:49.722 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.720+0000 7f90fa7fc640 1 -- 172.21.15.97:0/351899156 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 233+0+0 (unknown 1586942737 0 0) 0x7f90e8011900 con 0x7f910414be90 2024-03-25T18:20:49.723 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.720+0000 7f90fa7fc640 1 -- 172.21.15.97:0/351899156 >> v1:172.21.15.97:6790/0 conn(0x7f91040b4d60 legacy=0x7f90e800c310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:49.723 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.720+0000 7f90fa7fc640 1 -- 172.21.15.97:0/351899156 >> v1:172.21.15.98:6789/0 conn(0x7f90e800da30 legacy=0x7f90e800fe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:49.723 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.720+0000 7f90fa7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:20:49.723 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.721+0000 7f90fa7fc640 1 monclient: found mon.a 2024-03-25T18:20:49.723 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.721+0000 7f90fa7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:49.723 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.721+0000 7f90fa7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:20:49.723 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.721+0000 7f90fa7fc640 1 -- 172.21.15.97:0/351899156 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+,osdmap=0}) v3 -- 0x7f90e8014de0 con 0x7f910414be90 2024-03-25T18:20:49.723 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.721+0000 7f90fa7fc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:49.723 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.721+0000 7f90fa7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:20:49.723 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.721+0000 7f90fa7fc640 1 -- 172.21.15.97:0/351899156 --> v1:172.21.15.97:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91041d8750 con 0x7f910414be90 2024-03-25T18:20:49.723 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.721+0000 7f90fa7fc640 10 monclient: _finish_auth 0 2024-03-25T18:20:49.723 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.721+0000 7f90fa7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:20:49.723 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.721+0000 7f90fa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:49.724 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.721+0000 7f90fa7fc640 1 -- 172.21.15.97:0/351899156 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f9100004c80 con 0x7f910414be90 2024-03-25T18:20:49.724 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.721+0000 7f90fa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:49.724 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.721+0000 7f90fa7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:20:49.724 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:49.721+0000 7f90fa7fc640 10 monclient: dump: 2024-03-25T18:20:49.724 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:20:49.724 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:20:49.724 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:20:49.724 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:20:49.724 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:49.724 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:20:49.724 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:20:49.724 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:20:49.724 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:20:49.724 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:20:50.022 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:50.020+0000 7f90fa7fc640 1 -- 172.21.15.97:0/351899156 <== mon.0 v1:172.21.15.97:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9100004e50 con 0x7f910414be90 2024-03-25T18:20:50.022 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:50.020+0000 7f90fa7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:50.022 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:50.020+0000 7f90fbfff640 4 set_mon_vals no callback set 2024-03-25T18:20:50.023 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:50.021+0000 7f90fa7fc640 1 -- 172.21.15.97:0/351899156 <== mon.0 v1:172.21.15.97:6789/0 5 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2377622137 0 0) 0x7f910001aeb0 con 0x7f910414be90 2024-03-25T18:20:50.023 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:50.022+0000 7f91092ff640 10 monclient: get_auth_request con 0x7f91040b1730 auth_method 0 2024-03-25T18:20:50.024 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:50.022+0000 7f90fa7fc640 10 client.4426.objecter ms_handle_connect 0x7f91040b1730 2024-03-25T18:20:50.028 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:50.027+0000 7f90fa7fc640 1 -- 172.21.15.97:0/351899156 <== mon.0 v1:172.21.15.97:6789/0 6 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 430711918 0 0) 0x7f9100054ab0 con 0x7f910414be90 2024-03-25T18:20:50.029 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:50.027+0000 7f90fa7fc640 10 client.4426.objecter ms_dispatch 0x7f9104000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:20:50.029 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:50.027+0000 7f90fa7fc640 3 client.4426.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:20:50.029 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:50.027+0000 7f90fa7fc640 3 client.4426.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:20:50.029 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:50.027+0000 7f90fa7fc640 20 client.4426.objecter dump_active .. 0 homeless 2024-03-25T18:20:50.029 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:50.027+0000 7f90fa7fc640 1 -- 172.21.15.97:0/351899156 <== mon.0 v1:172.21.15.97:6789/0 7 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f91000503f0 con 0x7f910414be90 2024-03-25T18:20:50.029 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:50.027+0000 7f90fa7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:50.029 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:50.027+0000 7f90fa7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:20:50.258 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:50.256+0000 7f910bd64640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:50.258 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:50.256+0000 7f910bd64640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:50.258 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:50.256+0000 7f910bd64640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:20:50.258 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:50.256+0000 7f910bd64640 1 -- 172.21.15.97:0/351899156 --> v1:172.21.15.97:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f91040627b0 con 0x7f910414be90 2024-03-25T18:20:50.277 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:50.276+0000 7f90fa7fc640 1 -- 172.21.15.97:0/351899156 <== mon.0 v1:172.21.15.97:6789/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2169427412) 0x7f91000524f0 con 0x7f910414be90 2024-03-25T18:20:50.277 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:50.276+0000 7f90fa7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:50.277 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:50.276+0000 7f90fa7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:20:50.277 INFO:teuthology.misc.health.smithi097.stdout: 2024-03-25T18:20:50.277 INFO:teuthology.misc.health.smithi097.stdout:{"epoch":10,"fsid":"4c3d472c-79b8-486b-9e3d-2738d4306754","created":"2024-03-25T18:16:14.737309+0000","modified":"2024-03-25T18:16:20.357492+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:20.357492+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cb5152b9-262f-45c3-94b8-18d0e81df558","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6daf8793-0838-473d-a7d6-9a0d2894c75d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"136f1493-3b9c-4f43-a2e7-641482d69dae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"397d538c-4f64-4daf-9e8b-662d09f91327","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4f1df08-663f-4bda-93f8-3878ab7c24b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c4b18d70-a3ce-4b15-9b70-eae5525bd761","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8dd4033a-de0e-4f8a-96dc-59f3c1f825b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0077722e-ffa6-4a3d-9891-929d9869b286","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:20:50.280 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:50.278+0000 7f910bd64640 10 librados: watch_flush enter 2024-03-25T18:20:50.280 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:50.278+0000 7f910bd64640 10 librados: watch_flush exit 2024-03-25T18:20:50.280 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:50.278+0000 7f910bd64640 20 client.4426.objecter shutdown clearing up homeless session... 2024-03-25T18:20:50.280 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:50.278+0000 7f910bd64640 10 client.4426.objecter successfully canceled tick 2024-03-25T18:20:50.280 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:50.278+0000 7f910bd64640 1 -- 172.21.15.97:0/351899156 >> v1:172.21.15.98:6800/800387573 conn(0x7f91040b1730 legacy=0x7f90e807ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:50.280 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:50.278+0000 7f910bd64640 10 monclient: shutdown 2024-03-25T18:20:50.280 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:50.278+0000 7f910bd64640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:50.280 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:50.278+0000 7f910bd64640 1 -- 172.21.15.97:0/351899156 >> v1:172.21.15.97:6789/0 conn(0x7f910414be90 legacy=0x7f90e80063f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:50.280 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:50.278+0000 7f910bd64640 1 -- 172.21.15.97:0/351899156 shutdown_connections 2024-03-25T18:20:50.280 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:50.278+0000 7f910bd64640 1 -- 172.21.15.97:0/351899156 >> 172.21.15.97:0/351899156 conn(0x7f91040617a0 msgr2=0x7f910413f430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:50.281 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:50.278+0000 7f910bd64640 1 -- 172.21.15.97:0/351899156 shutdown_connections 2024-03-25T18:20:50.281 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:50.278+0000 7f910bd64640 1 -- 172.21.15.97:0/351899156 wait complete. 2024-03-25T18:20:50.281 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:50.278+0000 7f910bd64640 1 librados: shutdown 2024-03-25T18:20:50.291 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:20:52.638 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:04:54.677 34931== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:52.696 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:04:54.740 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:52.698 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:04:54.741 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:52.709 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:04:54.748 34931== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:52.710 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:04:54.753 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:52.754 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:04:54.792 34931== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:52.770 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:04:54.809 34935== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:52.781 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:04:54.820 34935== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:52.814 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:04:54.853 34935== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:57.097 DEBUG:teuthology.orchestra.run.smithi097:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:20:57.203 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.202+0000 7fbe3ecae640 10 monclient: get_monmap_and_config 2024-03-25T18:20:57.203 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.202+0000 7fbe3ecae640 10 monclient: build_initial_monmap 2024-03-25T18:20:57.203 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.202+0000 7fbe3ecae640 10 monclient: monmap: 2024-03-25T18:20:57.203 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:20:57.203 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:20:57.203 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:20:57.203337+0000 2024-03-25T18:20:57.203 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:20:57.203337+0000 2024-03-25T18:20:57.203 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:57.204 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:20:57.204 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:20:57.204 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T18:20:57.204 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-b 2024-03-25T18:20:57.204 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:20:57.205 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.203+0000 7fbe3ecae640 1 Processor -- start 2024-03-25T18:20:57.205 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.203+0000 7fbe3ecae640 1 -- start start 2024-03-25T18:20:57.205 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.203+0000 7fbe3ecae640 10 monclient: init 2024-03-25T18:20:57.205 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.203+0000 7fbe3ecae640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:57.205 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.203+0000 7fbe3ecae640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:20:57.206 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.203+0000 7fbe3ecae640 10 monclient(hunting): picked mon.noname-c con 0x7fbe3806ab80 addr v1:172.21.15.98:6789/0 2024-03-25T18:20:57.206 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.203+0000 7fbe3ecae640 10 monclient(hunting): picked mon.noname-b con 0x7fbe3806af60 addr v1:172.21.15.97:6790/0 2024-03-25T18:20:57.206 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.204+0000 7fbe3ecae640 10 monclient(hunting): picked mon.noname-a con 0x7fbe380b00b0 addr v1:172.21.15.97:6789/0 2024-03-25T18:20:57.206 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.204+0000 7fbe3ecae640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe380b0500 con 0x7fbe380b00b0 2024-03-25T18:20:57.206 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.204+0000 7fbe3ecae640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe3813ffa0 con 0x7fbe3806ab80 2024-03-25T18:20:57.206 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.204+0000 7fbe3ecae640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe380af520 con 0x7fbe3806af60 2024-03-25T18:20:57.206 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.204+0000 7fbe3ecae640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:57.206 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.204+0000 7fbe3ecae640 10 monclient(hunting): authenticate will time out at 7869.480044s 2024-03-25T18:20:57.216 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.215+0000 7fbe3ca4a640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fbe3806ab80 0x7fbe380ad880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.97:50578/0 (socket says 172.21.15.97:50578) 2024-03-25T18:20:57.216 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.215+0000 7fbe3ca4a640 1 -- 172.21.15.97:0/1784466708 learned_addr learned my addr 172.21.15.97:0/1784466708 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:20:57.232 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.231+0000 7fbe377fe640 1 -- 172.21.15.97:0/1784466708 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 950331618 0 0) 0x7fbe3813ffa0 con 0x7fbe3806ab80 2024-03-25T18:20:57.232 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.231+0000 7fbe377fe640 10 monclient(hunting): _negotiate 2024-03-25T18:20:57.232 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.231+0000 7fbe377fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:57.232 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.231+0000 7fbe377fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:57.232 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.231+0000 7fbe377fe640 10 monclient(hunting): my global_id is 4429 2024-03-25T18:20:57.233 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.231+0000 7fbe377fe640 1 -- 172.21.15.97:0/1784466708 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe28003520 con 0x7fbe3806ab80 2024-03-25T18:20:57.234 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.233+0000 7fbe377fe640 1 -- 172.21.15.97:0/1784466708 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3669518107 0 0) 0x7fbe380b0500 con 0x7fbe380b00b0 2024-03-25T18:20:57.234 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.233+0000 7fbe377fe640 10 monclient(hunting): _negotiate 2024-03-25T18:20:57.234 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.233+0000 7fbe377fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:57.234 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.233+0000 7fbe377fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:57.234 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.233+0000 7fbe377fe640 10 monclient(hunting): my global_id is 4431 2024-03-25T18:20:57.234 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.233+0000 7fbe377fe640 1 -- 172.21.15.97:0/1784466708 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe3813ffa0 con 0x7fbe380b00b0 2024-03-25T18:20:57.241 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.239+0000 7fbe377fe640 1 -- 172.21.15.97:0/1784466708 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2984791666 0 0) 0x7fbe380af520 con 0x7fbe3806af60 2024-03-25T18:20:57.241 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.239+0000 7fbe377fe640 10 monclient(hunting): _negotiate 2024-03-25T18:20:57.241 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.239+0000 7fbe377fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:57.241 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.239+0000 7fbe377fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:57.241 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.239+0000 7fbe377fe640 10 monclient(hunting): my global_id is 4421 2024-03-25T18:20:57.241 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.239+0000 7fbe377fe640 1 -- 172.21.15.97:0/1784466708 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe380b0500 con 0x7fbe3806af60 2024-03-25T18:20:57.245 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.243+0000 7fbe377fe640 1 -- 172.21.15.97:0/1784466708 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1153094456 0 0) 0x7fbe28003520 con 0x7fbe3806ab80 2024-03-25T18:20:57.245 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.243+0000 7fbe377fe640 1 -- 172.21.15.97:0/1784466708 >> v1:172.21.15.97:6790/0 conn(0x7fbe3806af60 legacy=0x7fbe3813f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:57.245 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.243+0000 7fbe377fe640 1 -- 172.21.15.97:0/1784466708 >> v1:172.21.15.97:6789/0 conn(0x7fbe380b00b0 legacy=0x7fbe3814dcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:57.245 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.243+0000 7fbe377fe640 10 monclient: _finish_hunting 0 2024-03-25T18:20:57.245 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.243+0000 7fbe377fe640 1 monclient: found mon.noname-c 2024-03-25T18:20:57.245 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.243+0000 7fbe377fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:57.245 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.243+0000 7fbe377fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.98:6789/0 2024-03-25T18:20:57.245 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.243+0000 7fbe377fe640 1 -- 172.21.15.97:0/1784466708 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe38072890 con 0x7fbe3806ab80 2024-03-25T18:20:57.245 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.243+0000 7fbe377fe640 10 monclient: _finish_auth 0 2024-03-25T18:20:57.245 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.243+0000 7fbe377fe640 10 monclient: _check_auth_tickets 2024-03-25T18:20:57.245 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.243+0000 7fbe377fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:57.245 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.243+0000 7fbe377fe640 1 -- 172.21.15.97:0/1784466708 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7fbe2c001690 con 0x7fbe3806ab80 2024-03-25T18:20:57.245 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.243+0000 7fbe3ecae640 5 monclient: authenticate success, global_id 4429 2024-03-25T18:20:57.246 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.243+0000 7fbe3ecae640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:20:57.246 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.243+0000 7fbe377fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:57.246 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.243+0000 7fbe377fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:20:57.246 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.243+0000 7fbe377fe640 10 monclient: dump: 2024-03-25T18:20:57.246 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:20:57.246 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:20:57.246 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:20:57.246 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:20:57.246 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:57.246 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:20:57.246 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:20:57.246 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:20:57.247 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:20:57.247 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:20:57.247 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.243+0000 7fbe3ecae640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:20:57.262 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.260+0000 7fbe377fe640 1 -- 172.21.15.97:0/1784466708 <== mon.1 v1:172.21.15.98:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbe2c0017f0 con 0x7fbe3806ab80 2024-03-25T18:20:57.262 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.260+0000 7fbe377fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:57.262 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.260+0000 7fbe377fe640 1 -- 172.21.15.97:0/1784466708 <== mon.1 v1:172.21.15.98:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7fbe2c001c00 con 0x7fbe3806ab80 2024-03-25T18:20:57.262 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.260+0000 7fbe377fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:57.262 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.260+0000 7fbe377fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:20:57.262 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.260+0000 7fbe377fe640 10 monclient: dump: 2024-03-25T18:20:57.262 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:20:57.262 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:20:57.263 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:20:57.263 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:20:57.263 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:57.263 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:20:57.263 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:20:57.263 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:20:57.263 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:20:57.263 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:20:57.263 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.260+0000 7fbe3ecae640 10 monclient: get_monmap_and_config success 2024-03-25T18:20:57.263 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.260+0000 7fbe3ecae640 4 set_mon_vals no callback set 2024-03-25T18:20:57.265 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.261+0000 7fbe3ecae640 10 monclient: shutdown 2024-03-25T18:20:57.265 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.261+0000 7fbe3ecae640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:57.265 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.261+0000 7fbe3ecae640 1 -- 172.21.15.97:0/1784466708 >> v1:172.21.15.98:6789/0 conn(0x7fbe3806ab80 legacy=0x7fbe380ad880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:57.265 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.261+0000 7fbe3ecae640 1 -- 172.21.15.97:0/1784466708 shutdown_connections 2024-03-25T18:20:57.265 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.261+0000 7fbe3ecae640 1 -- 172.21.15.97:0/1784466708 >> 172.21.15.97:0/1784466708 conn(0x7fbe380b4a80 msgr2=0x7fbe380b4e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:57.266 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.261+0000 7fbe3ecae640 1 -- 172.21.15.97:0/1784466708 shutdown_connections 2024-03-25T18:20:57.266 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.261+0000 7fbe3ecae640 1 -- 172.21.15.97:0/1784466708 wait complete. 2024-03-25T18:20:57.266 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.262+0000 7fbe3ecae640 10 monclient: build_initial_monmap 2024-03-25T18:20:57.266 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.262+0000 7fbe3ecae640 10 monclient: monmap: 2024-03-25T18:20:57.266 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:20:57.266 INFO:teuthology.misc.health.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:20:57.266 INFO:teuthology.misc.health.smithi097.stderr:last_changed 0.000000 2024-03-25T18:20:57.266 INFO:teuthology.misc.health.smithi097.stderr:created 0.000000 2024-03-25T18:20:57.266 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:57.266 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 1 2024-03-25T18:20:57.266 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:20:57.266 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T18:20:57.267 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-c 2024-03-25T18:20:57.267 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:20:57.267 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.262+0000 7fbe3ecae640 1 librados: starting msgr at 2024-03-25T18:20:57.267 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.262+0000 7fbe3ecae640 1 librados: starting objecter 2024-03-25T18:20:57.267 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.262+0000 7fbe3ecae640 1 Processor -- start 2024-03-25T18:20:57.267 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.263+0000 7fbe3ecae640 1 -- start start 2024-03-25T18:20:57.267 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.263+0000 7fbe3ecae640 1 librados: setting wanted keys 2024-03-25T18:20:57.267 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.263+0000 7fbe3ecae640 1 librados: calling monclient init 2024-03-25T18:20:57.267 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.263+0000 7fbe3ecae640 10 monclient: init 2024-03-25T18:20:57.267 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.263+0000 7fbe3ecae640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:57.267 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.263+0000 7fbe3ecae640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:20:57.267 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.263+0000 7fbe3ecae640 10 monclient(hunting): picked mon.noname-a con 0x7fbe3806ab80 addr v1:172.21.15.97:6789/0 2024-03-25T18:20:57.267 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.263+0000 7fbe3ecae640 10 monclient(hunting): picked mon.noname-b con 0x7fbe380b00b0 addr v1:172.21.15.98:6789/0 2024-03-25T18:20:57.267 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.263+0000 7fbe3ecae640 10 monclient(hunting): picked mon.noname-c con 0x7fbe38145e80 addr v1:172.21.15.97:6790/0 2024-03-25T18:20:57.268 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.263+0000 7fbe3ecae640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe381456f0 con 0x7fbe3806ab80 2024-03-25T18:20:57.268 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.263+0000 7fbe3ecae640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe381daca0 con 0x7fbe380b00b0 2024-03-25T18:20:57.268 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.263+0000 7fbe3ecae640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe381da930 con 0x7fbe38145e80 2024-03-25T18:20:57.268 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.263+0000 7fbe3ecae640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:57.268 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.263+0000 7fbe3ecae640 10 monclient(hunting): authenticate will time out at 7869.539729s 2024-03-25T18:20:57.274 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.273+0000 7fbe3d24b640 1 --1- >> v1:172.21.15.97:6790/0 conn(0x7fbe38145e80 0x7fbe381d7ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.97:6790/0 says I am v1:172.21.15.97:58540/0 (socket says 172.21.15.97:58540) 2024-03-25T18:20:57.274 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.273+0000 7fbe3d24b640 1 -- 172.21.15.97:0/2719091754 learned_addr learned my addr 172.21.15.97:0/2719091754 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:20:57.275 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.274+0000 7fbe357fa640 10 client.?.objecter ms_handle_connect 0x7fbe38145e80 2024-03-25T18:20:57.275 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.274+0000 7fbe357fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:57.276 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.275+0000 7fbe357fa640 10 client.?.objecter ms_handle_connect 0x7fbe380b00b0 2024-03-25T18:20:57.277 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.275+0000 7fbe357fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:57.283 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.282+0000 7fbe357fa640 10 client.?.objecter ms_handle_connect 0x7fbe3806ab80 2024-03-25T18:20:57.283 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.282+0000 7fbe357fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:57.286 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.285+0000 7fbe357fa640 1 -- 172.21.15.97:0/2719091754 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1402707655 0 0) 0x7fbe381da930 con 0x7fbe38145e80 2024-03-25T18:20:57.286 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.285+0000 7fbe357fa640 10 monclient(hunting): _negotiate 2024-03-25T18:20:57.286 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.285+0000 7fbe357fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:57.286 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.285+0000 7fbe357fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:57.287 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.285+0000 7fbe357fa640 10 monclient(hunting): my global_id is 4424 2024-03-25T18:20:57.287 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.285+0000 7fbe357fa640 1 -- 172.21.15.97:0/2719091754 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe1c004eb0 con 0x7fbe38145e80 2024-03-25T18:20:57.289 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.288+0000 7fbe357fa640 1 -- 172.21.15.97:0/2719091754 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3524914564 0 0) 0x7fbe381daca0 con 0x7fbe380b00b0 2024-03-25T18:20:57.290 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.288+0000 7fbe357fa640 10 monclient(hunting): _negotiate 2024-03-25T18:20:57.290 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.288+0000 7fbe357fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:57.290 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.288+0000 7fbe357fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:57.290 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.288+0000 7fbe357fa640 10 monclient(hunting): my global_id is 4432 2024-03-25T18:20:57.290 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.288+0000 7fbe357fa640 1 -- 172.21.15.97:0/2719091754 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe381da930 con 0x7fbe380b00b0 2024-03-25T18:20:57.300 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.298+0000 7fbe357fa640 1 -- 172.21.15.97:0/2719091754 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 591721803 0 0) 0x7fbe381456f0 con 0x7fbe3806ab80 2024-03-25T18:20:57.300 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.298+0000 7fbe357fa640 10 monclient(hunting): _negotiate 2024-03-25T18:20:57.300 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.298+0000 7fbe357fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:57.300 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.298+0000 7fbe357fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:57.300 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.298+0000 7fbe357fa640 10 monclient(hunting): my global_id is 4434 2024-03-25T18:20:57.300 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.298+0000 7fbe357fa640 1 -- 172.21.15.97:0/2719091754 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe381daca0 con 0x7fbe3806ab80 2024-03-25T18:20:57.300 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.298+0000 7fbe357fa640 1 -- 172.21.15.97:0/2719091754 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3004820773 0 0) 0x7fbe381da930 con 0x7fbe380b00b0 2024-03-25T18:20:57.300 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.298+0000 7fbe357fa640 1 -- 172.21.15.97:0/2719091754 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbe381456f0 con 0x7fbe380b00b0 2024-03-25T18:20:57.300 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.298+0000 7fbe357fa640 1 -- 172.21.15.97:0/2719091754 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7fbe24001870 con 0x7fbe380b00b0 2024-03-25T18:20:57.300 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.298+0000 7fbe357fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:57.301 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.298+0000 7fbe357fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:20:57.301 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.298+0000 7fbe357fa640 10 monclient(hunting): dump: 2024-03-25T18:20:57.301 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:20:57.301 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:20:57.301 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:20:57.301 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:20:57.301 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:57.301 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:20:57.301 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:20:57.301 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:20:57.301 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:20:57.301 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:20:57.301 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.298+0000 7fbe357fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:20:57.310 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.308+0000 7fbe357fa640 1 -- 172.21.15.97:0/2719091754 <== mon.1 v1:172.21.15.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 648447563 0 0) 0x7fbe381456f0 con 0x7fbe380b00b0 2024-03-25T18:20:57.310 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.308+0000 7fbe357fa640 1 -- 172.21.15.97:0/2719091754 >> v1:172.21.15.97:6790/0 conn(0x7fbe38145e80 legacy=0x7fbe381d7ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:57.310 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.308+0000 7fbe357fa640 1 -- 172.21.15.97:0/2719091754 >> v1:172.21.15.97:6789/0 conn(0x7fbe3806ab80 legacy=0x7fbe381db1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:57.311 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.308+0000 7fbe357fa640 10 monclient: _finish_hunting 0 2024-03-25T18:20:57.311 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.308+0000 7fbe357fa640 1 monclient: found mon.b 2024-03-25T18:20:57.311 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.308+0000 7fbe357fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:57.311 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.308+0000 7fbe357fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:20:57.311 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.308+0000 7fbe357fa640 1 -- 172.21.15.97:0/2719091754 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe381d81d0 con 0x7fbe380b00b0 2024-03-25T18:20:57.311 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.308+0000 7fbe357fa640 10 monclient: _finish_auth 0 2024-03-25T18:20:57.311 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.308+0000 7fbe357fa640 10 monclient: _check_auth_tickets 2024-03-25T18:20:57.311 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.308+0000 7fbe357fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:57.311 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.308+0000 7fbe3ecae640 5 monclient: authenticate success, global_id 4432 2024-03-25T18:20:57.311 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.308+0000 7fbe3ecae640 10 monclient: _renew_subs 2024-03-25T18:20:57.311 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.308+0000 7fbe3ecae640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:20:57.311 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.308+0000 7fbe3ecae640 1 -- 172.21.15.97:0/2719091754 --> v1:172.21.15.98:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe38142d80 con 0x7fbe380b00b0 2024-03-25T18:20:57.311 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.308+0000 7fbe3ecae640 10 client.4432.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:20:57.311 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.308+0000 7fbe3ecae640 10 monclient: _renew_subs 2024-03-25T18:20:57.312 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.308+0000 7fbe3ecae640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:20:57.312 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.308+0000 7fbe3ecae640 1 -- 172.21.15.97:0/2719091754 --> v1:172.21.15.98:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe381e3ce0 con 0x7fbe380b00b0 2024-03-25T18:20:57.312 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.308+0000 7fbe3ecae640 1 librados: init done 2024-03-25T18:20:57.312 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.309+0000 7fbe3ecae640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:57.312 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.309+0000 7fbe3ecae640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:57.312 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.309+0000 7fbe3ecae640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:20:57.313 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.309+0000 7fbe3ecae640 1 -- 172.21.15.97:0/2719091754 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe381daac0 con 0x7fbe380b00b0 2024-03-25T18:20:57.579 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.577+0000 7fbe357fa640 1 -- 172.21.15.97:0/2719091754 <== mon.1 v1:172.21.15.98:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbe24003610 con 0x7fbe380b00b0 2024-03-25T18:20:57.579 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.577+0000 7fbe357fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:57.579 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.577+0000 7fbe357fa640 1 -- 172.21.15.97:0/2719091754 <== mon.1 v1:172.21.15.98:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7fbe24001d80 con 0x7fbe380b00b0 2024-03-25T18:20:57.579 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.577+0000 7fbe357fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:57.579 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.577+0000 7fbe357fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:20:57.579 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.577+0000 7fbe357fa640 10 monclient: dump: 2024-03-25T18:20:57.579 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:20:57.579 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:20:57.579 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:20:57.579 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:20:57.579 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:57.580 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:20:57.580 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:20:57.580 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:20:57.580 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:20:57.580 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:20:57.580 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.577+0000 7fbe36ffd640 4 set_mon_vals no callback set 2024-03-25T18:20:57.580 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.578+0000 7fbe357fa640 1 -- 172.21.15.97:0/2719091754 <== mon.1 v1:172.21.15.98:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2377622137 0 0) 0x7fbe24003b20 con 0x7fbe380b00b0 2024-03-25T18:20:57.581 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.579+0000 7fbe3ca4a640 10 monclient: get_auth_request con 0x7fbe1c06de50 auth_method 0 2024-03-25T18:20:57.581 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.579+0000 7fbe357fa640 10 client.4432.objecter ms_handle_connect 0x7fbe1c06de50 2024-03-25T18:20:57.585 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.583+0000 7fbe357fa640 1 -- 172.21.15.97:0/2719091754 <== mon.1 v1:172.21.15.98:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 430711918 0 0) 0x7fbe24053590 con 0x7fbe380b00b0 2024-03-25T18:20:57.585 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.583+0000 7fbe357fa640 10 client.4432.objecter ms_dispatch 0x7fbe38000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:20:57.585 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.583+0000 7fbe357fa640 3 client.4432.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:20:57.585 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.583+0000 7fbe357fa640 3 client.4432.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:20:57.585 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.583+0000 7fbe357fa640 20 client.4432.objecter dump_active .. 0 homeless 2024-03-25T18:20:57.585 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.584+0000 7fbe357fa640 1 -- 172.21.15.97:0/2719091754 <== mon.1 v1:172.21.15.98:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbe2404f0a0 con 0x7fbe380b00b0 2024-03-25T18:20:57.586 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.584+0000 7fbe357fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:57.586 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.584+0000 7fbe357fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:20:57.835 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.833+0000 7fbe3ecae640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:57.835 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.834+0000 7fbe3ecae640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:57.835 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.834+0000 7fbe3ecae640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:20:57.835 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.834+0000 7fbe3ecae640 1 -- 172.21.15.97:0/2719091754 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbe380627b0 con 0x7fbe380b00b0 2024-03-25T18:20:57.871 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.870+0000 7fbe357fa640 1 -- 172.21.15.97:0/2719091754 <== mon.1 v1:172.21.15.98:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2169427412) 0x7fbe24050fd0 con 0x7fbe380b00b0 2024-03-25T18:20:57.871 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.870+0000 7fbe357fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:57.871 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.870+0000 7fbe357fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:20:57.871 INFO:teuthology.misc.health.smithi097.stdout: 2024-03-25T18:20:57.872 INFO:teuthology.misc.health.smithi097.stdout:{"epoch":10,"fsid":"4c3d472c-79b8-486b-9e3d-2738d4306754","created":"2024-03-25T18:16:14.737309+0000","modified":"2024-03-25T18:16:20.357492+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:20.357492+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cb5152b9-262f-45c3-94b8-18d0e81df558","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6daf8793-0838-473d-a7d6-9a0d2894c75d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"136f1493-3b9c-4f43-a2e7-641482d69dae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"397d538c-4f64-4daf-9e8b-662d09f91327","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4f1df08-663f-4bda-93f8-3878ab7c24b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c4b18d70-a3ce-4b15-9b70-eae5525bd761","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8dd4033a-de0e-4f8a-96dc-59f3c1f825b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0077722e-ffa6-4a3d-9891-929d9869b286","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:20:57.874 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.872+0000 7fbe3ecae640 10 librados: watch_flush enter 2024-03-25T18:20:57.874 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.872+0000 7fbe3ecae640 10 librados: watch_flush exit 2024-03-25T18:20:57.874 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.872+0000 7fbe3ecae640 20 client.4432.objecter shutdown clearing up homeless session... 2024-03-25T18:20:57.874 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.872+0000 7fbe3ecae640 10 client.4432.objecter successfully canceled tick 2024-03-25T18:20:57.874 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.872+0000 7fbe3ecae640 1 -- 172.21.15.97:0/2719091754 >> v1:172.21.15.98:6800/800387573 conn(0x7fbe1c06de50 legacy=0x7fbe1c070310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:57.874 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.872+0000 7fbe3ecae640 10 monclient: shutdown 2024-03-25T18:20:57.874 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.872+0000 7fbe3ecae640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:57.875 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.872+0000 7fbe3ecae640 1 -- 172.21.15.97:0/2719091754 >> v1:172.21.15.98:6789/0 conn(0x7fbe380b00b0 legacy=0x7fbe381d73b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:57.875 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.872+0000 7fbe3ecae640 1 -- 172.21.15.97:0/2719091754 shutdown_connections 2024-03-25T18:20:57.875 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.872+0000 7fbe3ecae640 1 -- 172.21.15.97:0/2719091754 >> 172.21.15.97:0/2719091754 conn(0x7fbe380b4a80 msgr2=0x7fbe3806f070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:57.875 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.872+0000 7fbe3ecae640 1 -- 172.21.15.97:0/2719091754 shutdown_connections 2024-03-25T18:20:57.875 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.872+0000 7fbe3ecae640 1 -- 172.21.15.97:0/2719091754 wait complete. 2024-03-25T18:20:57.875 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:20:57.872+0000 7fbe3ecae640 1 librados: shutdown 2024-03-25T18:20:57.885 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:21:04.692 DEBUG:teuthology.orchestra.run.smithi097:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:21:04.802 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.801+0000 7f34dd064640 10 monclient: get_monmap_and_config 2024-03-25T18:21:04.803 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.801+0000 7f34dd064640 10 monclient: build_initial_monmap 2024-03-25T18:21:04.803 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.801+0000 7f34dd064640 10 monclient: monmap: 2024-03-25T18:21:04.803 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:21:04.803 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:21:04.803 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:21:04.802903+0000 2024-03-25T18:21:04.803 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:21:04.802903+0000 2024-03-25T18:21:04.803 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:21:04.803 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:21:04.803 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:21:04.803 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T18:21:04.803 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-b 2024-03-25T18:21:04.803 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:21:04.805 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.802+0000 7f34dd064640 1 Processor -- start 2024-03-25T18:21:04.805 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.802+0000 7f34dd064640 1 -- start start 2024-03-25T18:21:04.805 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.802+0000 7f34dd064640 10 monclient: init 2024-03-25T18:21:04.805 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.802+0000 7f34dd064640 10 monclient: _reopen_session rank -1 2024-03-25T18:21:04.805 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.802+0000 7f34dd064640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:21:04.805 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.802+0000 7f34dd064640 10 monclient(hunting): picked mon.noname-b con 0x7f34d80a98f0 addr v1:172.21.15.97:6790/0 2024-03-25T18:21:04.805 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.802+0000 7f34dd064640 10 monclient(hunting): picked mon.noname-a con 0x7f34d813f710 addr v1:172.21.15.97:6789/0 2024-03-25T18:21:04.805 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.802+0000 7f34dd064640 10 monclient(hunting): picked mon.noname-c con 0x7f34d81430c0 addr v1:172.21.15.98:6789/0 2024-03-25T18:21:04.805 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.802+0000 7f34dd064640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34d8142520 con 0x7f34d813f710 2024-03-25T18:21:04.805 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.802+0000 7f34dd064640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34d80b28a0 con 0x7f34d81430c0 2024-03-25T18:21:04.805 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.802+0000 7f34dd064640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34d80b2a30 con 0x7f34d80a98f0 2024-03-25T18:21:04.805 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.802+0000 7f34dd064640 10 monclient(hunting): _renew_subs 2024-03-25T18:21:04.806 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.802+0000 7f34dd064640 10 monclient(hunting): authenticate will time out at 7877.078807s 2024-03-25T18:21:04.819 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.818+0000 7f34d6d9d640 1 --1- >> v1:172.21.15.97:6790/0 conn(0x7f34d80a98f0 0x7f34d80a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6790/0 says I am v1:172.21.15.97:56658/0 (socket says 172.21.15.97:56658) 2024-03-25T18:21:04.819 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.818+0000 7f34d6d9d640 1 -- 172.21.15.97:0/867740241 learned_addr learned my addr 172.21.15.97:0/867740241 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:21:04.832 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.830+0000 7f34d5d9b640 1 -- 172.21.15.97:0/867740241 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1274792173 0 0) 0x7f34d80b28a0 con 0x7f34d81430c0 2024-03-25T18:21:04.832 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.830+0000 7f34d5d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:21:04.832 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.830+0000 7f34d5d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:04.832 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.830+0000 7f34d5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:04.832 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.830+0000 7f34d5d9b640 10 monclient(hunting): my global_id is 4435 2024-03-25T18:21:04.832 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.830+0000 7f34d5d9b640 1 -- 172.21.15.97:0/867740241 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34c0003520 con 0x7f34d81430c0 2024-03-25T18:21:04.832 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.830+0000 7f34d5d9b640 1 -- 172.21.15.97:0/867740241 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2238177737 0 0) 0x7f34d8142520 con 0x7f34d813f710 2024-03-25T18:21:04.832 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.830+0000 7f34d5d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:21:04.832 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.830+0000 7f34d5d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:04.832 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.830+0000 7f34d5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:04.832 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.830+0000 7f34d5d9b640 10 monclient(hunting): my global_id is 4437 2024-03-25T18:21:04.832 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.830+0000 7f34d5d9b640 1 -- 172.21.15.97:0/867740241 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34d80b28a0 con 0x7f34d813f710 2024-03-25T18:21:04.833 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.831+0000 7f34d5d9b640 1 -- 172.21.15.97:0/867740241 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2500801870 0 0) 0x7f34d80b2a30 con 0x7f34d80a98f0 2024-03-25T18:21:04.833 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.831+0000 7f34d5d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:21:04.833 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.831+0000 7f34d5d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:04.833 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.831+0000 7f34d5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:04.833 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.831+0000 7f34d5d9b640 10 monclient(hunting): my global_id is 4427 2024-03-25T18:21:04.833 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.831+0000 7f34d5d9b640 1 -- 172.21.15.97:0/867740241 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34d8142520 con 0x7f34d80a98f0 2024-03-25T18:21:04.845 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.843+0000 7f34d5d9b640 1 -- 172.21.15.97:0/867740241 <== mon.2 v1:172.21.15.97:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3334726254 0 0) 0x7f34d8142520 con 0x7f34d80a98f0 2024-03-25T18:21:04.845 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.843+0000 7f34d5d9b640 1 -- 172.21.15.97:0/867740241 >> v1:172.21.15.98:6789/0 conn(0x7f34d81430c0 legacy=0x7f34d814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:04.845 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.843+0000 7f34d5d9b640 1 -- 172.21.15.97:0/867740241 >> v1:172.21.15.97:6789/0 conn(0x7f34d813f710 legacy=0x7f34d81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:04.845 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.843+0000 7f34d5d9b640 10 monclient: _finish_hunting 0 2024-03-25T18:21:04.845 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.843+0000 7f34d5d9b640 1 monclient: found mon.noname-b 2024-03-25T18:21:04.845 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.843+0000 7f34d5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:21:04.845 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.843+0000 7f34d5d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.97:6790/0 2024-03-25T18:21:04.845 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.843+0000 7f34d5d9b640 1 -- 172.21.15.97:0/867740241 --> v1:172.21.15.97:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34d80b44c0 con 0x7f34d80a98f0 2024-03-25T18:21:04.845 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.843+0000 7f34d5d9b640 10 monclient: _finish_auth 0 2024-03-25T18:21:04.845 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.843+0000 7f34d5d9b640 10 monclient: _check_auth_tickets 2024-03-25T18:21:04.845 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.843+0000 7f34d5d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:21:04.845 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.843+0000 7f34d5d9b640 1 -- 172.21.15.97:0/867740241 <== mon.2 v1:172.21.15.97:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f34c80015b0 con 0x7f34d80a98f0 2024-03-25T18:21:04.845 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.843+0000 7f34d5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:04.845 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.843+0000 7f34d5d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:21:04.845 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.843+0000 7f34d5d9b640 10 monclient: dump: 2024-03-25T18:21:04.846 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:21:04.846 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:21:04.846 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:21:04.846 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:21:04.846 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:04.846 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:21:04.846 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:21:04.846 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:21:04.846 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:21:04.846 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:21:04.846 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.843+0000 7f34dd064640 5 monclient: authenticate success, global_id 4427 2024-03-25T18:21:04.846 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.843+0000 7f34dd064640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:21:04.850 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.849+0000 7f34d5d9b640 1 -- 172.21.15.97:0/867740241 <== mon.2 v1:172.21.15.97:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f34c8001710 con 0x7f34d80a98f0 2024-03-25T18:21:04.850 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.849+0000 7f34d5d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:21:04.850 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.849+0000 7f34dd064640 10 monclient: get_monmap_and_config success 2024-03-25T18:21:04.850 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.849+0000 7f34dd064640 4 set_mon_vals no callback set 2024-03-25T18:21:04.851 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.849+0000 7f34dd064640 10 monclient: shutdown 2024-03-25T18:21:04.851 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.849+0000 7f34dd064640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:21:04.851 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.849+0000 7f34dd064640 1 -- 172.21.15.97:0/867740241 >> v1:172.21.15.97:6790/0 conn(0x7f34d80a98f0 legacy=0x7f34d80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:04.851 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.850+0000 7f34dd064640 1 -- 172.21.15.97:0/867740241 shutdown_connections 2024-03-25T18:21:04.851 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.850+0000 7f34dd064640 1 -- 172.21.15.97:0/867740241 >> 172.21.15.97:0/867740241 conn(0x7f34d80b7ea0 msgr2=0x7f34d80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:21:04.852 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.850+0000 7f34dd064640 1 -- 172.21.15.97:0/867740241 shutdown_connections 2024-03-25T18:21:04.852 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.850+0000 7f34dd064640 1 -- 172.21.15.97:0/867740241 wait complete. 2024-03-25T18:21:04.852 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.850+0000 7f34dd064640 10 monclient: build_initial_monmap 2024-03-25T18:21:04.852 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.850+0000 7f34dd064640 10 monclient: monmap: 2024-03-25T18:21:04.853 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:21:04.853 INFO:teuthology.misc.health.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:21:04.853 INFO:teuthology.misc.health.smithi097.stderr:last_changed 0.000000 2024-03-25T18:21:04.853 INFO:teuthology.misc.health.smithi097.stderr:created 0.000000 2024-03-25T18:21:04.853 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:21:04.853 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 1 2024-03-25T18:21:04.853 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:21:04.853 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T18:21:04.853 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-c 2024-03-25T18:21:04.853 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:21:04.853 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.851+0000 7f34dd064640 1 librados: starting msgr at 2024-03-25T18:21:04.853 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.851+0000 7f34dd064640 1 librados: starting objecter 2024-03-25T18:21:04.853 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.851+0000 7f34dd064640 1 Processor -- start 2024-03-25T18:21:04.853 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.851+0000 7f34dd064640 1 -- start start 2024-03-25T18:21:04.853 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.851+0000 7f34dd064640 1 librados: setting wanted keys 2024-03-25T18:21:04.854 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.851+0000 7f34dd064640 1 librados: calling monclient init 2024-03-25T18:21:04.854 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.851+0000 7f34dd064640 10 monclient: init 2024-03-25T18:21:04.855 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.852+0000 7f34dd064640 10 monclient: _reopen_session rank -1 2024-03-25T18:21:04.855 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.852+0000 7f34dd064640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:21:04.855 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.852+0000 7f34dd064640 10 monclient(hunting): picked mon.noname-a con 0x7f34d80a98f0 addr v1:172.21.15.97:6789/0 2024-03-25T18:21:04.855 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.852+0000 7f34dd064640 10 monclient(hunting): picked mon.noname-b con 0x7f34d813f710 addr v1:172.21.15.98:6789/0 2024-03-25T18:21:04.855 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.852+0000 7f34dd064640 10 monclient(hunting): picked mon.noname-c con 0x7f34d81430c0 addr v1:172.21.15.97:6790/0 2024-03-25T18:21:04.855 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.852+0000 7f34dd064640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34d81d8080 con 0x7f34d80a98f0 2024-03-25T18:21:04.855 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.852+0000 7f34dd064640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34d80b1770 con 0x7f34d813f710 2024-03-25T18:21:04.855 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.852+0000 7f34dd064640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34d80b09d0 con 0x7f34d81430c0 2024-03-25T18:21:04.855 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.852+0000 7f34dd064640 10 monclient(hunting): _renew_subs 2024-03-25T18:21:04.855 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.852+0000 7f34dd064640 10 monclient(hunting): authenticate will time out at 7877.128010s 2024-03-25T18:21:04.869 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.867+0000 7f34d6d9d640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f34d80a98f0 0x7f34d81d7740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:49280/0 (socket says 172.21.15.97:49280) 2024-03-25T18:21:04.869 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.867+0000 7f34d6d9d640 1 -- 172.21.15.97:0/2395700157 learned_addr learned my addr 172.21.15.97:0/2395700157 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:21:04.873 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.871+0000 7f34bf7fe640 10 client.?.objecter ms_handle_connect 0x7f34d81430c0 2024-03-25T18:21:04.873 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.871+0000 7f34bf7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:04.876 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.875+0000 7f34bf7fe640 10 client.?.objecter ms_handle_connect 0x7f34d80a98f0 2024-03-25T18:21:04.876 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.875+0000 7f34bf7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:04.883 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.882+0000 7f34bf7fe640 10 client.?.objecter ms_handle_connect 0x7f34d813f710 2024-03-25T18:21:04.883 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.882+0000 7f34bf7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:04.893 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.891+0000 7f34bf7fe640 1 -- 172.21.15.97:0/2395700157 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1038596559 0 0) 0x7f34d80b1770 con 0x7f34d813f710 2024-03-25T18:21:04.893 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.891+0000 7f34bf7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:21:04.893 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.891+0000 7f34bf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:04.893 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.891+0000 7f34bf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:04.893 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.891+0000 7f34bf7fe640 10 monclient(hunting): my global_id is 4438 2024-03-25T18:21:04.893 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.891+0000 7f34bf7fe640 1 -- 172.21.15.97:0/2395700157 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34c4004eb0 con 0x7f34d813f710 2024-03-25T18:21:04.894 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.892+0000 7f34bf7fe640 1 -- 172.21.15.97:0/2395700157 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3073165944 0 0) 0x7f34d80b09d0 con 0x7f34d81430c0 2024-03-25T18:21:04.894 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.892+0000 7f34bf7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:21:04.894 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.892+0000 7f34bf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:04.894 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.892+0000 7f34bf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:04.894 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.892+0000 7f34bf7fe640 10 monclient(hunting): my global_id is 4430 2024-03-25T18:21:04.894 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.892+0000 7f34bf7fe640 1 -- 172.21.15.97:0/2395700157 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34d80b1770 con 0x7f34d81430c0 2024-03-25T18:21:04.894 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.893+0000 7f34bf7fe640 1 -- 172.21.15.97:0/2395700157 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4072102391 0 0) 0x7f34d81d8080 con 0x7f34d80a98f0 2024-03-25T18:21:04.894 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.893+0000 7f34bf7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:21:04.895 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.893+0000 7f34bf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:04.895 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.893+0000 7f34bf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:04.895 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.893+0000 7f34bf7fe640 10 monclient(hunting): my global_id is 4440 2024-03-25T18:21:04.895 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.893+0000 7f34bf7fe640 1 -- 172.21.15.97:0/2395700157 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34d80b09d0 con 0x7f34d80a98f0 2024-03-25T18:21:04.908 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.906+0000 7f34bf7fe640 1 -- 172.21.15.97:0/2395700157 <== mon.2 v1:172.21.15.97:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4009608602 0 0) 0x7f34d80b1770 con 0x7f34d81430c0 2024-03-25T18:21:04.908 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.907+0000 7f34bf7fe640 1 -- 172.21.15.97:0/2395700157 --> v1:172.21.15.97:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f34d81d8080 con 0x7f34d81430c0 2024-03-25T18:21:04.908 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.907+0000 7f34bf7fe640 1 -- 172.21.15.97:0/2395700157 <== mon.2 v1:172.21.15.97:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f34d0001a70 con 0x7f34d81430c0 2024-03-25T18:21:04.908 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.907+0000 7f34bf7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:04.908 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.907+0000 7f34bf7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:21:04.908 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.907+0000 7f34bf7fe640 10 monclient(hunting): dump: 2024-03-25T18:21:04.909 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:21:04.909 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:21:04.909 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:21:04.909 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:21:04.909 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:04.909 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:21:04.909 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:21:04.909 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:21:04.909 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:21:04.909 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:21:04.909 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.907+0000 7f34bf7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:21:04.913 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.911+0000 7f34bf7fe640 1 -- 172.21.15.97:0/2395700157 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2595045605 0 0) 0x7f34d80b09d0 con 0x7f34d80a98f0 2024-03-25T18:21:04.913 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.911+0000 7f34bf7fe640 1 -- 172.21.15.97:0/2395700157 --> v1:172.21.15.97:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f34d80b1770 con 0x7f34d80a98f0 2024-03-25T18:21:04.913 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.911+0000 7f34bf7fe640 1 -- 172.21.15.97:0/2395700157 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f34c8001c00 con 0x7f34d80a98f0 2024-03-25T18:21:04.913 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.911+0000 7f34bf7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:04.913 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.911+0000 7f34bf7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:21:04.913 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.911+0000 7f34bf7fe640 10 monclient(hunting): dump: 2024-03-25T18:21:04.913 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:21:04.913 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:21:04.913 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:21:04.913 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:21:04.914 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:04.914 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:21:04.914 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:21:04.914 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:21:04.914 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:21:04.914 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:21:04.914 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.913+0000 7f34bf7fe640 1 -- 172.21.15.97:0/2395700157 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2848604143 0 0) 0x7f34c4004eb0 con 0x7f34d813f710 2024-03-25T18:21:04.915 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.913+0000 7f34bf7fe640 1 -- 172.21.15.97:0/2395700157 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f34d80b09d0 con 0x7f34d813f710 2024-03-25T18:21:04.915 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.913+0000 7f34bf7fe640 1 -- 172.21.15.97:0/2395700157 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f34cc001930 con 0x7f34d813f710 2024-03-25T18:21:04.915 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.913+0000 7f34bf7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:04.915 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.913+0000 7f34bf7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:21:04.915 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.913+0000 7f34bf7fe640 10 monclient(hunting): dump: 2024-03-25T18:21:04.915 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:21:04.915 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:21:04.915 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:21:04.915 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:21:04.915 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:04.915 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:21:04.915 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:21:04.915 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:21:04.915 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:21:04.915 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:21:04.927 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.925+0000 7f34bf7fe640 1 -- 172.21.15.97:0/2395700157 <== mon.2 v1:172.21.15.97:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3351556771 0 0) 0x7f34d81d8080 con 0x7f34d81430c0 2024-03-25T18:21:04.927 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.925+0000 7f34bf7fe640 1 -- 172.21.15.97:0/2395700157 >> v1:172.21.15.98:6789/0 conn(0x7f34d813f710 legacy=0x7f34d80b1060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:04.927 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.925+0000 7f34bf7fe640 1 -- 172.21.15.97:0/2395700157 >> v1:172.21.15.97:6789/0 conn(0x7f34d80a98f0 legacy=0x7f34d81d7740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:04.927 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.925+0000 7f34bf7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:21:04.927 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.925+0000 7f34bf7fe640 1 monclient: found mon.c 2024-03-25T18:21:04.927 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.925+0000 7f34bf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:21:04.927 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.925+0000 7f34bf7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.97:6790/0 2024-03-25T18:21:04.927 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.925+0000 7f34bf7fe640 1 -- 172.21.15.97:0/2395700157 --> v1:172.21.15.97:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34d80b0c40 con 0x7f34d81430c0 2024-03-25T18:21:04.927 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.925+0000 7f34bf7fe640 10 monclient: _finish_auth 0 2024-03-25T18:21:04.927 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.925+0000 7f34bf7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:21:04.927 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.925+0000 7f34bf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:21:04.927 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.925+0000 7f34dd064640 5 monclient: authenticate success, global_id 4430 2024-03-25T18:21:04.928 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.925+0000 7f34dd064640 10 monclient: _renew_subs 2024-03-25T18:21:04.928 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.925+0000 7f34dd064640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.97:6790/0 2024-03-25T18:21:04.928 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.925+0000 7f34dd064640 1 -- 172.21.15.97:0/2395700157 --> v1:172.21.15.97:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34d80ad420 con 0x7f34d81430c0 2024-03-25T18:21:04.928 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.925+0000 7f34dd064640 10 client.4430.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:21:04.928 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.925+0000 7f34dd064640 10 monclient: _renew_subs 2024-03-25T18:21:04.928 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.925+0000 7f34dd064640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.97:6790/0 2024-03-25T18:21:04.928 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.925+0000 7f34dd064640 1 -- 172.21.15.97:0/2395700157 --> v1:172.21.15.97:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f34d80ae960 con 0x7f34d81430c0 2024-03-25T18:21:04.928 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.925+0000 7f34dd064640 1 librados: init done 2024-03-25T18:21:04.928 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.926+0000 7f34dd064640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:04.929 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.926+0000 7f34dd064640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:04.929 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.926+0000 7f34dd064640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.97:6790/0 2024-03-25T18:21:04.929 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:04.926+0000 7f34dd064640 1 -- 172.21.15.97:0/2395700157 --> v1:172.21.15.97:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34d80627b0 con 0x7f34d81430c0 2024-03-25T18:21:05.240 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:05.239+0000 7f34bf7fe640 1 -- 172.21.15.97:0/2395700157 <== mon.2 v1:172.21.15.97:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f34d0001f80 con 0x7f34d81430c0 2024-03-25T18:21:05.240 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:05.239+0000 7f34bf7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:21:05.240 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:05.239+0000 7f34bf7fe640 1 -- 172.21.15.97:0/2395700157 <== mon.2 v1:172.21.15.97:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f34d0003b10 con 0x7f34d81430c0 2024-03-25T18:21:05.241 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:05.239+0000 7f34bf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:05.241 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:05.239+0000 7f34bf7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:21:05.241 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:05.239+0000 7f34bf7fe640 10 monclient: dump: 2024-03-25T18:21:05.241 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:21:05.241 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:21:05.241 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:21:05.241 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:21:05.241 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:05.241 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:21:05.241 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:21:05.241 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:21:05.241 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:21:05.241 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:21:05.241 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:05.239+0000 7f34d559a640 4 set_mon_vals no callback set 2024-03-25T18:21:05.242 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:05.240+0000 7f34bf7fe640 1 -- 172.21.15.97:0/2395700157 <== mon.2 v1:172.21.15.97:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2377622137 0 0) 0x7f34d0019ae0 con 0x7f34d81430c0 2024-03-25T18:21:05.242 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:05.240+0000 7f34d6d9d640 10 monclient: get_auth_request con 0x7f34c406e060 auth_method 0 2024-03-25T18:21:05.242 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:05.241+0000 7f34bf7fe640 10 client.4430.objecter ms_handle_connect 0x7f34c406e060 2024-03-25T18:21:05.246 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:05.245+0000 7f34bf7fe640 1 -- 172.21.15.97:0/2395700157 <== mon.2 v1:172.21.15.97:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 430711918 0 0) 0x7f34d0053770 con 0x7f34d81430c0 2024-03-25T18:21:05.246 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:05.245+0000 7f34bf7fe640 10 client.4430.objecter ms_dispatch 0x7f34d8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:21:05.246 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:05.245+0000 7f34bf7fe640 3 client.4430.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:21:05.247 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:05.245+0000 7f34bf7fe640 3 client.4430.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:21:05.247 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:05.245+0000 7f34bf7fe640 20 client.4430.objecter dump_active .. 0 homeless 2024-03-25T18:21:05.247 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:05.245+0000 7f34bf7fe640 1 -- 172.21.15.97:0/2395700157 <== mon.2 v1:172.21.15.97:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f34d004f0b0 con 0x7f34d81430c0 2024-03-25T18:21:05.247 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:05.245+0000 7f34bf7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:05.247 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:05.245+0000 7f34bf7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:21:05.492 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:05.491+0000 7f34dd064640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:05.493 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:05.491+0000 7f34dd064640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:05.493 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:05.491+0000 7f34dd064640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.97:6790/0 2024-03-25T18:21:05.493 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:05.491+0000 7f34dd064640 1 -- 172.21.15.97:0/2395700157 --> v1:172.21.15.97:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f34d8069f40 con 0x7f34d81430c0 2024-03-25T18:21:05.521 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:05.519+0000 7f34bf7fe640 1 -- 172.21.15.97:0/2395700157 <== mon.2 v1:172.21.15.97:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2169427412) 0x7f34d00511b0 con 0x7f34d81430c0 2024-03-25T18:21:05.521 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:05.519+0000 7f34bf7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:05.521 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:05.519+0000 7f34bf7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:21:05.521 INFO:teuthology.misc.health.smithi097.stdout: 2024-03-25T18:21:05.521 INFO:teuthology.misc.health.smithi097.stdout:{"epoch":10,"fsid":"4c3d472c-79b8-486b-9e3d-2738d4306754","created":"2024-03-25T18:16:14.737309+0000","modified":"2024-03-25T18:16:20.357492+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:20.357492+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cb5152b9-262f-45c3-94b8-18d0e81df558","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6daf8793-0838-473d-a7d6-9a0d2894c75d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"136f1493-3b9c-4f43-a2e7-641482d69dae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"397d538c-4f64-4daf-9e8b-662d09f91327","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4f1df08-663f-4bda-93f8-3878ab7c24b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c4b18d70-a3ce-4b15-9b70-eae5525bd761","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8dd4033a-de0e-4f8a-96dc-59f3c1f825b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0077722e-ffa6-4a3d-9891-929d9869b286","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:21:05.523 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:05.522+0000 7f34dd064640 10 librados: watch_flush enter 2024-03-25T18:21:05.524 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:05.522+0000 7f34dd064640 10 librados: watch_flush exit 2024-03-25T18:21:05.524 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:05.522+0000 7f34dd064640 20 client.4430.objecter shutdown clearing up homeless session... 2024-03-25T18:21:05.524 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:05.522+0000 7f34dd064640 10 client.4430.objecter successfully canceled tick 2024-03-25T18:21:05.524 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:05.522+0000 7f34dd064640 1 -- 172.21.15.97:0/2395700157 >> v1:172.21.15.98:6800/800387573 conn(0x7f34c406e060 legacy=0x7f34c4070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:05.524 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:05.522+0000 7f34dd064640 10 monclient: shutdown 2024-03-25T18:21:05.524 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:05.522+0000 7f34dd064640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:21:05.524 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:05.522+0000 7f34dd064640 1 -- 172.21.15.97:0/2395700157 >> v1:172.21.15.97:6790/0 conn(0x7f34d81430c0 legacy=0x7f34d80acd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:05.524 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:05.522+0000 7f34dd064640 1 -- 172.21.15.97:0/2395700157 shutdown_connections 2024-03-25T18:21:05.524 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:05.522+0000 7f34dd064640 1 -- 172.21.15.97:0/2395700157 >> 172.21.15.97:0/2395700157 conn(0x7f34d80b7ea0 msgr2=0x7f34d8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:21:05.524 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:05.522+0000 7f34dd064640 1 -- 172.21.15.97:0/2395700157 shutdown_connections 2024-03-25T18:21:05.524 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:05.522+0000 7f34dd064640 1 -- 172.21.15.97:0/2395700157 wait complete. 2024-03-25T18:21:05.524 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:05.522+0000 7f34dd064640 1 librados: shutdown 2024-03-25T18:21:05.536 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:21:12.342 DEBUG:teuthology.orchestra.run.smithi097:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:21:12.453 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.452+0000 7efdc6b63640 10 monclient: get_monmap_and_config 2024-03-25T18:21:12.453 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.452+0000 7efdc6b63640 10 monclient: build_initial_monmap 2024-03-25T18:21:12.454 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.452+0000 7efdc6b63640 10 monclient: monmap: 2024-03-25T18:21:12.454 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:21:12.454 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:21:12.454 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:21:12.453677+0000 2024-03-25T18:21:12.454 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:21:12.453677+0000 2024-03-25T18:21:12.454 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:21:12.454 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:21:12.454 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:21:12.454 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T18:21:12.454 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-b 2024-03-25T18:21:12.454 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:21:12.455 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.453+0000 7efdc6b63640 1 Processor -- start 2024-03-25T18:21:12.456 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.453+0000 7efdc6b63640 1 -- start start 2024-03-25T18:21:12.456 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.453+0000 7efdc6b63640 10 monclient: init 2024-03-25T18:21:12.456 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.454+0000 7efdc6b63640 10 monclient: _reopen_session rank -1 2024-03-25T18:21:12.456 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.454+0000 7efdc6b63640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:21:12.456 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.454+0000 7efdc6b63640 10 monclient(hunting): picked mon.noname-a con 0x7efdc00a98f0 addr v1:172.21.15.97:6789/0 2024-03-25T18:21:12.456 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.454+0000 7efdc6b63640 10 monclient(hunting): picked mon.noname-c con 0x7efdc013f710 addr v1:172.21.15.98:6789/0 2024-03-25T18:21:12.456 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.454+0000 7efdc6b63640 10 monclient(hunting): picked mon.noname-b con 0x7efdc01430c0 addr v1:172.21.15.97:6790/0 2024-03-25T18:21:12.456 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.454+0000 7efdc6b63640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdc0142520 con 0x7efdc00a98f0 2024-03-25T18:21:12.456 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.454+0000 7efdc6b63640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdc00b28a0 con 0x7efdc013f710 2024-03-25T18:21:12.456 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.454+0000 7efdc6b63640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdc00b2a30 con 0x7efdc01430c0 2024-03-25T18:21:12.456 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.454+0000 7efdc6b63640 10 monclient(hunting): _renew_subs 2024-03-25T18:21:12.456 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.454+0000 7efdc6b63640 10 monclient(hunting): authenticate will time out at 7884.730037s 2024-03-25T18:21:12.462 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.461+0000 7efdbffff640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7efdc013f710 0x7efdc01429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.97:60408/0 (socket says 172.21.15.97:60408) 2024-03-25T18:21:12.463 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.461+0000 7efdbffff640 1 -- 172.21.15.97:0/3426555041 learned_addr learned my addr 172.21.15.97:0/3426555041 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:21:12.473 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.472+0000 7efdbf7fe640 1 -- 172.21.15.97:0/3426555041 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1835530856 0 0) 0x7efdc00b28a0 con 0x7efdc013f710 2024-03-25T18:21:12.474 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.472+0000 7efdbf7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:21:12.474 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.472+0000 7efdbf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:12.474 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.472+0000 7efdbf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:12.474 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.472+0000 7efdbf7fe640 10 monclient(hunting): my global_id is 4441 2024-03-25T18:21:12.474 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.472+0000 7efdbf7fe640 1 -- 172.21.15.97:0/3426555041 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efdb0003520 con 0x7efdc013f710 2024-03-25T18:21:12.479 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.477+0000 7efdbf7fe640 1 -- 172.21.15.97:0/3426555041 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3972578664 0 0) 0x7efdc00b2a30 con 0x7efdc01430c0 2024-03-25T18:21:12.479 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.477+0000 7efdbf7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:21:12.479 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.477+0000 7efdbf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:12.479 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.477+0000 7efdbf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:12.479 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.477+0000 7efdbf7fe640 10 monclient(hunting): my global_id is 4433 2024-03-25T18:21:12.479 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.478+0000 7efdbf7fe640 1 -- 172.21.15.97:0/3426555041 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efdc00b28a0 con 0x7efdc01430c0 2024-03-25T18:21:12.483 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.482+0000 7efdbf7fe640 1 -- 172.21.15.97:0/3426555041 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1321426140 0 0) 0x7efdb0003520 con 0x7efdc013f710 2024-03-25T18:21:12.484 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.482+0000 7efdbf7fe640 1 -- 172.21.15.97:0/3426555041 >> v1:172.21.15.97:6790/0 conn(0x7efdc01430c0 legacy=0x7efdc014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:12.484 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.482+0000 7efdbf7fe640 1 -- 172.21.15.97:0/3426555041 >> v1:172.21.15.97:6789/0 conn(0x7efdc00a98f0 legacy=0x7efdc00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:12.484 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.482+0000 7efdbf7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:21:12.484 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.482+0000 7efdbf7fe640 1 monclient: found mon.noname-c 2024-03-25T18:21:12.484 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.482+0000 7efdbf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:21:12.484 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.482+0000 7efdbf7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.98:6789/0 2024-03-25T18:21:12.484 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.482+0000 7efdbf7fe640 1 -- 172.21.15.97:0/3426555041 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdc00b44c0 con 0x7efdc013f710 2024-03-25T18:21:12.484 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.482+0000 7efdbf7fe640 10 monclient: _finish_auth 0 2024-03-25T18:21:12.484 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.482+0000 7efdbf7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:21:12.484 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.482+0000 7efdbf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:21:12.484 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.482+0000 7efdbf7fe640 1 -- 172.21.15.97:0/3426555041 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7efdac001690 con 0x7efdc013f710 2024-03-25T18:21:12.484 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.482+0000 7efdbf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:12.485 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.482+0000 7efdbf7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:21:12.485 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.482+0000 7efdbf7fe640 10 monclient: dump: 2024-03-25T18:21:12.485 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:21:12.485 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:21:12.485 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:21:12.485 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:21:12.485 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:12.485 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:21:12.485 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:21:12.485 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:21:12.485 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:21:12.485 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:21:12.486 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.482+0000 7efdc6b63640 5 monclient: authenticate success, global_id 4441 2024-03-25T18:21:12.486 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.482+0000 7efdc6b63640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:21:12.487 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.485+0000 7efdbf7fe640 1 -- 172.21.15.97:0/3426555041 <== mon.1 v1:172.21.15.98:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efdac0017f0 con 0x7efdc013f710 2024-03-25T18:21:12.487 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.485+0000 7efdbf7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:21:12.487 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.485+0000 7efdc6b63640 10 monclient: get_monmap_and_config success 2024-03-25T18:21:12.487 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.485+0000 7efdc6b63640 4 set_mon_vals no callback set 2024-03-25T18:21:12.489 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.486+0000 7efdbf7fe640 1 -- 172.21.15.97:0/3426555041 <== mon.1 v1:172.21.15.98:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7efdac001c00 con 0x7efdc013f710 2024-03-25T18:21:12.489 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.486+0000 7efdbf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:12.489 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.486+0000 7efdbf7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:21:12.489 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.486+0000 7efdbf7fe640 10 monclient: dump: 2024-03-25T18:21:12.489 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:21:12.489 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:21:12.489 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:21:12.490 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:21:12.490 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:12.490 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:21:12.490 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:21:12.490 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:21:12.490 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:21:12.490 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:21:12.490 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.486+0000 7efdc6b63640 10 monclient: shutdown 2024-03-25T18:21:12.490 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.486+0000 7efdc6b63640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:21:12.490 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.486+0000 7efdc6b63640 1 -- 172.21.15.97:0/3426555041 >> v1:172.21.15.98:6789/0 conn(0x7efdc013f710 legacy=0x7efdc01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:12.490 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.486+0000 7efdc6b63640 1 -- 172.21.15.97:0/3426555041 shutdown_connections 2024-03-25T18:21:12.490 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.486+0000 7efdc6b63640 1 -- 172.21.15.97:0/3426555041 >> 172.21.15.97:0/3426555041 conn(0x7efdc00b7ea0 msgr2=0x7efdc00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:21:12.490 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.486+0000 7efdc6b63640 1 -- 172.21.15.97:0/3426555041 shutdown_connections 2024-03-25T18:21:12.490 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.486+0000 7efdc6b63640 1 -- 172.21.15.97:0/3426555041 wait complete. 2024-03-25T18:21:12.491 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.487+0000 7efdc6b63640 10 monclient: build_initial_monmap 2024-03-25T18:21:12.491 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.487+0000 7efdc6b63640 10 monclient: monmap: 2024-03-25T18:21:12.491 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:21:12.491 INFO:teuthology.misc.health.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:21:12.491 INFO:teuthology.misc.health.smithi097.stderr:last_changed 0.000000 2024-03-25T18:21:12.491 INFO:teuthology.misc.health.smithi097.stderr:created 0.000000 2024-03-25T18:21:12.491 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:21:12.491 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 1 2024-03-25T18:21:12.491 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:21:12.491 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T18:21:12.491 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-c 2024-03-25T18:21:12.491 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:21:12.491 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.487+0000 7efdc6b63640 1 librados: starting msgr at 2024-03-25T18:21:12.491 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.487+0000 7efdc6b63640 1 librados: starting objecter 2024-03-25T18:21:12.492 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.487+0000 7efdc6b63640 1 Processor -- start 2024-03-25T18:21:12.492 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.487+0000 7efdc6b63640 1 -- start start 2024-03-25T18:21:12.492 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.487+0000 7efdc6b63640 1 librados: setting wanted keys 2024-03-25T18:21:12.492 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.487+0000 7efdc6b63640 1 librados: calling monclient init 2024-03-25T18:21:12.492 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.487+0000 7efdc6b63640 10 monclient: init 2024-03-25T18:21:12.493 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.488+0000 7efdc6b63640 10 monclient: _reopen_session rank -1 2024-03-25T18:21:12.493 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.488+0000 7efdc6b63640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:21:12.493 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.488+0000 7efdc6b63640 10 monclient(hunting): picked mon.noname-b con 0x7efdc00a98f0 addr v1:172.21.15.98:6789/0 2024-03-25T18:21:12.493 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.488+0000 7efdc6b63640 10 monclient(hunting): picked mon.noname-a con 0x7efdc013f710 addr v1:172.21.15.97:6789/0 2024-03-25T18:21:12.493 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.488+0000 7efdc6b63640 10 monclient(hunting): picked mon.noname-c con 0x7efdc01430c0 addr v1:172.21.15.97:6790/0 2024-03-25T18:21:12.493 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.488+0000 7efdc6b63640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdc00abca0 con 0x7efdc013f710 2024-03-25T18:21:12.493 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.488+0000 7efdc6b63640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdc01d7730 con 0x7efdc00a98f0 2024-03-25T18:21:12.494 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.488+0000 7efdc6b63640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdc01d41e0 con 0x7efdc01430c0 2024-03-25T18:21:12.494 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.488+0000 7efdc6b63640 10 monclient(hunting): _renew_subs 2024-03-25T18:21:12.494 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.488+0000 7efdc6b63640 10 monclient(hunting): authenticate will time out at 7884.764829s 2024-03-25T18:21:12.500 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.498+0000 7efdc48ff640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7efdc00a98f0 0x7efdc01d7c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.97:60446/0 (socket says 172.21.15.97:60446) 2024-03-25T18:21:12.500 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.498+0000 7efdc48ff640 1 -- 172.21.15.97:0/1469169023 learned_addr learned my addr 172.21.15.97:0/1469169023 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:21:12.505 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.504+0000 7efdbd7fa640 10 client.?.objecter ms_handle_connect 0x7efdc00a98f0 2024-03-25T18:21:12.505 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.504+0000 7efdbd7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:12.513 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.512+0000 7efdbd7fa640 10 client.?.objecter ms_handle_connect 0x7efdc013f710 2024-03-25T18:21:12.514 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.512+0000 7efdbd7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:12.514 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.513+0000 7efdbd7fa640 1 -- 172.21.15.97:0/1469169023 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1507521292 0 0) 0x7efdc01d7730 con 0x7efdc00a98f0 2024-03-25T18:21:12.514 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.513+0000 7efdbd7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:21:12.514 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.513+0000 7efdbd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:12.514 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.513+0000 7efdbd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:12.515 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.513+0000 7efdbd7fa640 10 monclient(hunting): my global_id is 4444 2024-03-25T18:21:12.515 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.513+0000 7efdbd7fa640 1 -- 172.21.15.97:0/1469169023 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efda4004eb0 con 0x7efdc00a98f0 2024-03-25T18:21:12.517 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.516+0000 7efdbd7fa640 10 client.?.objecter ms_handle_connect 0x7efdc01430c0 2024-03-25T18:21:12.517 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.516+0000 7efdbd7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:12.525 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.524+0000 7efdbd7fa640 1 -- 172.21.15.97:0/1469169023 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3996627476 0 0) 0x7efdc00abca0 con 0x7efdc013f710 2024-03-25T18:21:12.526 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.524+0000 7efdbd7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:21:12.526 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.524+0000 7efdbd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:12.526 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.524+0000 7efdbd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:12.526 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.524+0000 7efdbd7fa640 10 monclient(hunting): my global_id is 4446 2024-03-25T18:21:12.526 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.524+0000 7efdbd7fa640 1 -- 172.21.15.97:0/1469169023 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efdc01d7730 con 0x7efdc013f710 2024-03-25T18:21:12.534 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.533+0000 7efdbd7fa640 1 -- 172.21.15.97:0/1469169023 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 634252944 0 0) 0x7efdc01d41e0 con 0x7efdc01430c0 2024-03-25T18:21:12.534 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.533+0000 7efdbd7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:21:12.534 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.533+0000 7efdbd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:12.534 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.533+0000 7efdbd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:12.535 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.533+0000 7efdbd7fa640 10 monclient(hunting): my global_id is 4436 2024-03-25T18:21:12.535 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.533+0000 7efdbd7fa640 1 -- 172.21.15.97:0/1469169023 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efdc00abca0 con 0x7efdc01430c0 2024-03-25T18:21:12.538 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.536+0000 7efdbd7fa640 1 -- 172.21.15.97:0/1469169023 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3356827439 0 0) 0x7efda4004eb0 con 0x7efdc00a98f0 2024-03-25T18:21:12.538 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.536+0000 7efdbd7fa640 1 -- 172.21.15.97:0/1469169023 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efdc01d41e0 con 0x7efdc00a98f0 2024-03-25T18:21:12.538 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.536+0000 7efdbd7fa640 1 -- 172.21.15.97:0/1469169023 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7efdb4001a30 con 0x7efdc00a98f0 2024-03-25T18:21:12.538 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.536+0000 7efdbd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:12.538 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.536+0000 7efdbd7fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:21:12.538 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.536+0000 7efdbd7fa640 10 monclient(hunting): dump: 2024-03-25T18:21:12.538 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:21:12.538 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:21:12.538 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:21:12.538 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:21:12.538 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:12.538 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:21:12.538 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:21:12.539 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:21:12.539 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:21:12.539 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:21:12.539 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.536+0000 7efdbd7fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:21:12.562 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.560+0000 7efdbd7fa640 1 -- 172.21.15.97:0/1469169023 <== mon.1 v1:172.21.15.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3620996196 0 0) 0x7efdc01d41e0 con 0x7efdc00a98f0 2024-03-25T18:21:12.562 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.560+0000 7efdbd7fa640 1 -- 172.21.15.97:0/1469169023 >> v1:172.21.15.97:6790/0 conn(0x7efdc01430c0 legacy=0x7efdc01d4790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:12.562 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.560+0000 7efdbd7fa640 1 -- 172.21.15.97:0/1469169023 >> v1:172.21.15.97:6789/0 conn(0x7efdc013f710 legacy=0x7efdc0138640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:12.562 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.560+0000 7efdbd7fa640 10 monclient: _finish_hunting 0 2024-03-25T18:21:12.562 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.560+0000 7efdbd7fa640 1 monclient: found mon.b 2024-03-25T18:21:12.562 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.560+0000 7efdbd7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:21:12.562 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.560+0000 7efdbd7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:21:12.563 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.560+0000 7efdbd7fa640 1 -- 172.21.15.97:0/1469169023 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdc01d4370 con 0x7efdc00a98f0 2024-03-25T18:21:12.563 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.560+0000 7efdbd7fa640 10 monclient: _finish_auth 0 2024-03-25T18:21:12.563 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.560+0000 7efdbd7fa640 10 monclient: _check_auth_tickets 2024-03-25T18:21:12.563 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.560+0000 7efdbd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:21:12.563 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.560+0000 7efdc6b63640 5 monclient: authenticate success, global_id 4444 2024-03-25T18:21:12.563 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.560+0000 7efdc6b63640 10 monclient: _renew_subs 2024-03-25T18:21:12.563 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.560+0000 7efdc6b63640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:21:12.563 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.560+0000 7efdc6b63640 1 -- 172.21.15.97:0/1469169023 --> v1:172.21.15.98:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdc01d8320 con 0x7efdc00a98f0 2024-03-25T18:21:12.563 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.560+0000 7efdc6b63640 10 client.4444.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:21:12.563 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.560+0000 7efdc6b63640 10 monclient: _renew_subs 2024-03-25T18:21:12.563 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.560+0000 7efdc6b63640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:21:12.563 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.560+0000 7efdc6b63640 1 -- 172.21.15.97:0/1469169023 --> v1:172.21.15.98:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efdc0138f30 con 0x7efdc00a98f0 2024-03-25T18:21:12.563 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.560+0000 7efdc6b63640 1 librados: init done 2024-03-25T18:21:12.564 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.561+0000 7efdc6b63640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:12.564 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.561+0000 7efdc6b63640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:12.564 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.561+0000 7efdc6b63640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:21:12.564 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.561+0000 7efdc6b63640 1 -- 172.21.15.97:0/1469169023 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efdc00627b0 con 0x7efdc00a98f0 2024-03-25T18:21:12.872 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.871+0000 7efdbd7fa640 1 -- 172.21.15.97:0/1469169023 <== mon.1 v1:172.21.15.98:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efdb4001c70 con 0x7efdc00a98f0 2024-03-25T18:21:12.872 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.871+0000 7efdbd7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:21:12.872 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.871+0000 7efdbd7fa640 1 -- 172.21.15.97:0/1469169023 <== mon.1 v1:172.21.15.98:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7efdb4003a60 con 0x7efdc00a98f0 2024-03-25T18:21:12.872 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.871+0000 7efdbd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:12.873 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.871+0000 7efdbd7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:21:12.873 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.871+0000 7efdbd7fa640 10 monclient: dump: 2024-03-25T18:21:12.873 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:21:12.873 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:21:12.873 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:21:12.873 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:21:12.873 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:12.873 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:21:12.873 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:21:12.873 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:21:12.873 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:21:12.873 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:21:12.873 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.871+0000 7efdbeffd640 4 set_mon_vals no callback set 2024-03-25T18:21:12.874 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.871+0000 7efdbd7fa640 1 -- 172.21.15.97:0/1469169023 <== mon.1 v1:172.21.15.98:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2377622137 0 0) 0x7efdb4003c40 con 0x7efdc00a98f0 2024-03-25T18:21:12.874 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.872+0000 7efdbffff640 10 monclient: get_auth_request con 0x7efda406ddf0 auth_method 0 2024-03-25T18:21:12.874 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.872+0000 7efdbd7fa640 10 client.4444.objecter ms_handle_connect 0x7efda406ddf0 2024-03-25T18:21:12.880 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.878+0000 7efdbd7fa640 1 -- 172.21.15.97:0/1469169023 <== mon.1 v1:172.21.15.98:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 430711918 0 0) 0x7efdb40548b0 con 0x7efdc00a98f0 2024-03-25T18:21:12.880 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.878+0000 7efdbd7fa640 10 client.4444.objecter ms_dispatch 0x7efdc0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:21:12.880 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.878+0000 7efdbd7fa640 3 client.4444.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:21:12.880 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.878+0000 7efdbd7fa640 3 client.4444.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:21:12.880 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.879+0000 7efdbd7fa640 20 client.4444.objecter dump_active .. 0 homeless 2024-03-25T18:21:12.882 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.880+0000 7efdbd7fa640 1 -- 172.21.15.97:0/1469169023 <== mon.1 v1:172.21.15.98:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7efdb40503c0 con 0x7efdc00a98f0 2024-03-25T18:21:12.882 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.880+0000 7efdbd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:12.882 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:12.880+0000 7efdbd7fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:21:12.963 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:05:15.002 34931== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:12.977 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:05:15.016 34931== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:13.003 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:05:15.042 34931== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:13.029 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:05:15.072 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:13.039 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:05:15.083 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:13.055 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:05:15.094 34935== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:13.063 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:05:15.102 34935== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:13.079 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:05:15.122 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:13.097 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:05:15.136 34935== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:13.121 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:13.120+0000 7efdc6b63640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:13.121 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:13.120+0000 7efdc6b63640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:13.122 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:13.120+0000 7efdc6b63640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:21:13.122 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:13.120+0000 7efdc6b63640 1 -- 172.21.15.97:0/1469169023 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efdc013cf80 con 0x7efdc00a98f0 2024-03-25T18:21:13.211 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:13.209+0000 7efdbd7fa640 1 -- 172.21.15.97:0/1469169023 <== mon.1 v1:172.21.15.98:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2169427412) 0x7efdb40522f0 con 0x7efdc00a98f0 2024-03-25T18:21:13.211 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:13.209+0000 7efdbd7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:13.211 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:13.209+0000 7efdbd7fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:21:13.211 INFO:teuthology.misc.health.smithi097.stdout: 2024-03-25T18:21:13.211 INFO:teuthology.misc.health.smithi097.stdout:{"epoch":10,"fsid":"4c3d472c-79b8-486b-9e3d-2738d4306754","created":"2024-03-25T18:16:14.737309+0000","modified":"2024-03-25T18:16:20.357492+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:20.357492+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cb5152b9-262f-45c3-94b8-18d0e81df558","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6daf8793-0838-473d-a7d6-9a0d2894c75d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"136f1493-3b9c-4f43-a2e7-641482d69dae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"397d538c-4f64-4daf-9e8b-662d09f91327","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4f1df08-663f-4bda-93f8-3878ab7c24b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c4b18d70-a3ce-4b15-9b70-eae5525bd761","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8dd4033a-de0e-4f8a-96dc-59f3c1f825b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0077722e-ffa6-4a3d-9891-929d9869b286","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:21:13.214 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:13.212+0000 7efdc6b63640 10 librados: watch_flush enter 2024-03-25T18:21:13.214 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:13.212+0000 7efdc6b63640 10 librados: watch_flush exit 2024-03-25T18:21:13.214 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:13.212+0000 7efdc6b63640 20 client.4444.objecter shutdown clearing up homeless session... 2024-03-25T18:21:13.214 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:13.212+0000 7efdc6b63640 10 client.4444.objecter successfully canceled tick 2024-03-25T18:21:13.214 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:13.212+0000 7efdc6b63640 1 -- 172.21.15.97:0/1469169023 >> v1:172.21.15.98:6800/800387573 conn(0x7efda406ddf0 legacy=0x7efda40702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:13.214 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:13.212+0000 7efdc6b63640 10 monclient: shutdown 2024-03-25T18:21:13.214 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:13.212+0000 7efdc6b63640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:21:13.215 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:13.212+0000 7efdc6b63640 1 -- 172.21.15.97:0/1469169023 >> v1:172.21.15.98:6789/0 conn(0x7efdc00a98f0 legacy=0x7efdc01d7c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:13.215 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:13.212+0000 7efdc6b63640 1 -- 172.21.15.97:0/1469169023 shutdown_connections 2024-03-25T18:21:13.215 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:13.212+0000 7efdc6b63640 1 -- 172.21.15.97:0/1469169023 >> 172.21.15.97:0/1469169023 conn(0x7efdc00b7ea0 msgr2=0x7efdc0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:21:13.215 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:13.212+0000 7efdc6b63640 1 -- 172.21.15.97:0/1469169023 shutdown_connections 2024-03-25T18:21:13.215 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:13.212+0000 7efdc6b63640 1 -- 172.21.15.97:0/1469169023 wait complete. 2024-03-25T18:21:13.215 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:13.212+0000 7efdc6b63640 1 librados: shutdown 2024-03-25T18:21:13.224 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:21:20.031 DEBUG:teuthology.orchestra.run.smithi097:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:21:20.144 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.143+0000 7fd2efaa1640 10 monclient: get_monmap_and_config 2024-03-25T18:21:20.144 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.143+0000 7fd2efaa1640 10 monclient: build_initial_monmap 2024-03-25T18:21:20.144 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.143+0000 7fd2efaa1640 10 monclient: monmap: 2024-03-25T18:21:20.144 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:21:20.145 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:21:20.145 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:21:20.144495+0000 2024-03-25T18:21:20.145 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:21:20.144495+0000 2024-03-25T18:21:20.145 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:21:20.145 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:21:20.145 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:21:20.145 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T18:21:20.145 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-b 2024-03-25T18:21:20.145 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:21:20.146 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.144+0000 7fd2efaa1640 1 Processor -- start 2024-03-25T18:21:20.147 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.144+0000 7fd2efaa1640 1 -- start start 2024-03-25T18:21:20.147 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.144+0000 7fd2efaa1640 10 monclient: init 2024-03-25T18:21:20.147 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.144+0000 7fd2efaa1640 10 monclient: _reopen_session rank -1 2024-03-25T18:21:20.147 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.144+0000 7fd2efaa1640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:21:20.147 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.144+0000 7fd2efaa1640 10 monclient(hunting): picked mon.noname-c con 0x7fd2e80b4d60 addr v1:172.21.15.98:6789/0 2024-03-25T18:21:20.147 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.144+0000 7fd2efaa1640 10 monclient(hunting): picked mon.noname-b con 0x7fd2e80b1730 addr v1:172.21.15.97:6790/0 2024-03-25T18:21:20.147 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.145+0000 7fd2efaa1640 10 monclient(hunting): picked mon.noname-a con 0x7fd2e814be90 addr v1:172.21.15.97:6789/0 2024-03-25T18:21:20.147 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.145+0000 7fd2efaa1640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2e80af940 con 0x7fd2e814be90 2024-03-25T18:21:20.147 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.145+0000 7fd2efaa1640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2e80b0a70 con 0x7fd2e80b4d60 2024-03-25T18:21:20.147 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.145+0000 7fd2efaa1640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2e80b0d50 con 0x7fd2e80b1730 2024-03-25T18:21:20.147 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.145+0000 7fd2efaa1640 10 monclient(hunting): _renew_subs 2024-03-25T18:21:20.147 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.145+0000 7fd2efaa1640 10 monclient(hunting): authenticate will time out at 7892.421007s 2024-03-25T18:21:20.156 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.155+0000 7fd2ed03c640 1 --1- >> v1:172.21.15.97:6790/0 conn(0x7fd2e80b1730 0x7fd2e80b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6790/0 says I am v1:172.21.15.97:48812/0 (socket says 172.21.15.97:48812) 2024-03-25T18:21:20.156 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.155+0000 7fd2ed03c640 1 -- 172.21.15.97:0/4096698759 learned_addr learned my addr 172.21.15.97:0/4096698759 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:21:20.163 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.161+0000 7fd2ec83b640 1 -- 172.21.15.97:0/4096698759 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2807694452 0 0) 0x7fd2e80b0a70 con 0x7fd2e80b4d60 2024-03-25T18:21:20.163 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.161+0000 7fd2ec83b640 10 monclient(hunting): _negotiate 2024-03-25T18:21:20.163 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.161+0000 7fd2ec83b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:20.163 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.161+0000 7fd2ec83b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:20.163 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.161+0000 7fd2ec83b640 10 monclient(hunting): my global_id is 4447 2024-03-25T18:21:20.163 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.161+0000 7fd2ec83b640 1 -- 172.21.15.97:0/4096698759 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2d4003520 con 0x7fd2e80b4d60 2024-03-25T18:21:20.169 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.167+0000 7fd2ec83b640 1 -- 172.21.15.97:0/4096698759 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2548784301 0 0) 0x7fd2d4003520 con 0x7fd2e80b4d60 2024-03-25T18:21:20.169 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.167+0000 7fd2ec83b640 1 -- 172.21.15.97:0/4096698759 >> v1:172.21.15.97:6790/0 conn(0x7fd2e80b1730 legacy=0x7fd2e80b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:20.169 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.167+0000 7fd2ec83b640 1 -- 172.21.15.97:0/4096698759 >> v1:172.21.15.97:6789/0 conn(0x7fd2e814be90 legacy=0x7fd2e814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:21:20.169 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.167+0000 7fd2ec83b640 10 monclient: _finish_hunting 0 2024-03-25T18:21:20.169 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.167+0000 7fd2ec83b640 1 monclient: found mon.noname-c 2024-03-25T18:21:20.169 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.167+0000 7fd2ec83b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:21:20.169 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.167+0000 7fd2ec83b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.98:6789/0 2024-03-25T18:21:20.169 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.167+0000 7fd2ec83b640 1 -- 172.21.15.97:0/4096698759 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2e80b1030 con 0x7fd2e80b4d60 2024-03-25T18:21:20.169 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.167+0000 7fd2ec83b640 10 monclient: _finish_auth 0 2024-03-25T18:21:20.169 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.167+0000 7fd2ec83b640 10 monclient: _check_auth_tickets 2024-03-25T18:21:20.170 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.167+0000 7fd2ec83b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:21:20.170 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.167+0000 7fd2ec83b640 1 -- 172.21.15.97:0/4096698759 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7fd2e4001690 con 0x7fd2e80b4d60 2024-03-25T18:21:20.170 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.167+0000 7fd2efaa1640 5 monclient: authenticate success, global_id 4447 2024-03-25T18:21:20.170 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.167+0000 7fd2efaa1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:21:20.170 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.167+0000 7fd2ec83b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:20.170 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.167+0000 7fd2ec83b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:21:20.170 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.167+0000 7fd2ec83b640 10 monclient: dump: 2024-03-25T18:21:20.170 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:21:20.170 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:21:20.170 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:21:20.170 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:21:20.170 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:20.170 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:21:20.170 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:21:20.170 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:21:20.171 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:21:20.171 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:21:20.171 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.167+0000 7fd2efaa1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:21:20.181 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.179+0000 7fd2ec83b640 1 -- 172.21.15.97:0/4096698759 <== mon.1 v1:172.21.15.98:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd2e4001860 con 0x7fd2e80b4d60 2024-03-25T18:21:20.181 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.179+0000 7fd2ec83b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:21:20.181 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.179+0000 7fd2ec83b640 1 -- 172.21.15.97:0/4096698759 <== mon.1 v1:172.21.15.98:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7fd2e4001cb0 con 0x7fd2e80b4d60 2024-03-25T18:21:20.181 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.179+0000 7fd2efaa1640 10 monclient: get_monmap_and_config success 2024-03-25T18:21:20.181 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.179+0000 7fd2efaa1640 4 set_mon_vals no callback set 2024-03-25T18:21:20.181 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.179+0000 7fd2ec83b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:20.181 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.179+0000 7fd2ec83b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:21:20.181 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.179+0000 7fd2ec83b640 10 monclient: dump: 2024-03-25T18:21:20.181 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:21:20.181 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:21:20.181 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:21:20.181 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:21:20.181 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:20.182 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:21:20.182 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:21:20.182 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:21:20.182 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:21:20.182 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:21:20.184 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.180+0000 7fd2efaa1640 10 monclient: shutdown 2024-03-25T18:21:20.184 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.180+0000 7fd2efaa1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:21:20.184 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.180+0000 7fd2efaa1640 1 -- 172.21.15.97:0/4096698759 >> v1:172.21.15.98:6789/0 conn(0x7fd2e80b4d60 legacy=0x7fd2e813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:20.184 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.180+0000 7fd2efaa1640 1 -- 172.21.15.97:0/4096698759 shutdown_connections 2024-03-25T18:21:20.184 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.180+0000 7fd2efaa1640 1 -- 172.21.15.97:0/4096698759 >> 172.21.15.97:0/4096698759 conn(0x7fd2e80617a0 msgr2=0x7fd2e8061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:21:20.184 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.180+0000 7fd2efaa1640 1 -- 172.21.15.97:0/4096698759 shutdown_connections 2024-03-25T18:21:20.184 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.180+0000 7fd2efaa1640 1 -- 172.21.15.97:0/4096698759 wait complete. 2024-03-25T18:21:20.184 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.181+0000 7fd2efaa1640 10 monclient: build_initial_monmap 2024-03-25T18:21:20.184 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.181+0000 7fd2efaa1640 10 monclient: monmap: 2024-03-25T18:21:20.185 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:21:20.185 INFO:teuthology.misc.health.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:21:20.185 INFO:teuthology.misc.health.smithi097.stderr:last_changed 0.000000 2024-03-25T18:21:20.185 INFO:teuthology.misc.health.smithi097.stderr:created 0.000000 2024-03-25T18:21:20.185 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:21:20.185 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 1 2024-03-25T18:21:20.185 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:21:20.185 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T18:21:20.185 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-c 2024-03-25T18:21:20.185 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:21:20.185 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.181+0000 7fd2efaa1640 1 librados: starting msgr at 2024-03-25T18:21:20.185 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.181+0000 7fd2efaa1640 1 librados: starting objecter 2024-03-25T18:21:20.185 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.182+0000 7fd2efaa1640 1 Processor -- start 2024-03-25T18:21:20.185 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.182+0000 7fd2efaa1640 1 -- start start 2024-03-25T18:21:20.185 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.182+0000 7fd2efaa1640 1 librados: setting wanted keys 2024-03-25T18:21:20.186 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.182+0000 7fd2efaa1640 1 librados: calling monclient init 2024-03-25T18:21:20.186 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.182+0000 7fd2efaa1640 10 monclient: init 2024-03-25T18:21:20.186 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.182+0000 7fd2efaa1640 10 monclient: _reopen_session rank -1 2024-03-25T18:21:20.186 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.182+0000 7fd2efaa1640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:21:20.186 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.182+0000 7fd2efaa1640 10 monclient(hunting): picked mon.noname-b con 0x7fd2e80b1730 addr v1:172.21.15.98:6789/0 2024-03-25T18:21:20.186 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.182+0000 7fd2efaa1640 10 monclient(hunting): picked mon.noname-a con 0x7fd2e80b4d60 addr v1:172.21.15.97:6789/0 2024-03-25T18:21:20.186 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.182+0000 7fd2efaa1640 10 monclient(hunting): picked mon.noname-c con 0x7fd2e814be90 addr v1:172.21.15.97:6790/0 2024-03-25T18:21:20.186 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.182+0000 7fd2efaa1640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2e8142280 con 0x7fd2e80b4d60 2024-03-25T18:21:20.186 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.182+0000 7fd2efaa1640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2e8138180 con 0x7fd2e80b1730 2024-03-25T18:21:20.186 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.182+0000 7fd2efaa1640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2e8138460 con 0x7fd2e814be90 2024-03-25T18:21:20.186 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.182+0000 7fd2efaa1640 10 monclient(hunting): _renew_subs 2024-03-25T18:21:20.186 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.182+0000 7fd2efaa1640 10 monclient(hunting): authenticate will time out at 7892.458632s 2024-03-25T18:21:20.195 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.194+0000 7fd2ed83d640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fd2e80b1730 0x7fd2e8131e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.97:36626/0 (socket says 172.21.15.97:36626) 2024-03-25T18:21:20.195 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.194+0000 7fd2ed83d640 1 -- 172.21.15.97:0/850635344 learned_addr learned my addr 172.21.15.97:0/850635344 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:21:20.198 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.197+0000 7fd2de7fc640 10 client.?.objecter ms_handle_connect 0x7fd2e80b1730 2024-03-25T18:21:20.198 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.197+0000 7fd2de7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:20.204 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.203+0000 7fd2de7fc640 10 client.?.objecter ms_handle_connect 0x7fd2e814be90 2024-03-25T18:21:20.204 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.203+0000 7fd2de7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:20.212 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.211+0000 7fd2de7fc640 10 client.?.objecter ms_handle_connect 0x7fd2e80b4d60 2024-03-25T18:21:20.212 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.211+0000 7fd2de7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:20.216 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.215+0000 7fd2de7fc640 1 -- 172.21.15.97:0/850635344 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3265391772 0 0) 0x7fd2e8138180 con 0x7fd2e80b1730 2024-03-25T18:21:20.216 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.215+0000 7fd2de7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:21:20.216 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.215+0000 7fd2de7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:20.216 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.215+0000 7fd2de7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:20.216 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.215+0000 7fd2de7fc640 10 monclient(hunting): my global_id is 4450 2024-03-25T18:21:20.216 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.215+0000 7fd2de7fc640 1 -- 172.21.15.97:0/850635344 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2cc004eb0 con 0x7fd2e80b1730 2024-03-25T18:21:20.218 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.217+0000 7fd2de7fc640 1 -- 172.21.15.97:0/850635344 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2803649955 0 0) 0x7fd2e8142280 con 0x7fd2e80b4d60 2024-03-25T18:21:20.218 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.217+0000 7fd2de7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:21:20.218 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.217+0000 7fd2de7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:20.219 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.217+0000 7fd2de7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:20.219 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.217+0000 7fd2de7fc640 10 monclient(hunting): my global_id is 4449 2024-03-25T18:21:20.219 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.217+0000 7fd2de7fc640 1 -- 172.21.15.97:0/850635344 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2e8138180 con 0x7fd2e80b4d60 2024-03-25T18:21:20.219 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.217+0000 7fd2de7fc640 1 -- 172.21.15.97:0/850635344 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4173005088 0 0) 0x7fd2e8138460 con 0x7fd2e814be90 2024-03-25T18:21:20.219 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.217+0000 7fd2de7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:21:20.219 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.217+0000 7fd2de7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:20.219 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.217+0000 7fd2de7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:20.219 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.217+0000 7fd2de7fc640 10 monclient(hunting): my global_id is 4442 2024-03-25T18:21:20.219 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.217+0000 7fd2de7fc640 1 -- 172.21.15.97:0/850635344 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2e8142280 con 0x7fd2e814be90 2024-03-25T18:21:20.236 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.235+0000 7fd2de7fc640 1 -- 172.21.15.97:0/850635344 <== mon.2 v1:172.21.15.97:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4203584418 0 0) 0x7fd2e8142280 con 0x7fd2e814be90 2024-03-25T18:21:20.237 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.235+0000 7fd2de7fc640 1 -- 172.21.15.97:0/850635344 --> v1:172.21.15.97:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd2e8138460 con 0x7fd2e814be90 2024-03-25T18:21:20.237 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.235+0000 7fd2de7fc640 1 -- 172.21.15.97:0/850635344 <== mon.2 v1:172.21.15.97:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7fd2e0001a10 con 0x7fd2e814be90 2024-03-25T18:21:20.237 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.235+0000 7fd2de7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:20.237 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.235+0000 7fd2de7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:21:20.237 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.235+0000 7fd2de7fc640 10 monclient(hunting): dump: 2024-03-25T18:21:20.237 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:21:20.237 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:21:20.237 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:21:20.237 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:21:20.237 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:20.237 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:21:20.237 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:21:20.237 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:21:20.237 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:21:20.238 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:21:20.238 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.235+0000 7fd2de7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:21:20.238 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.235+0000 7fd2de7fc640 1 -- 172.21.15.97:0/850635344 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2973711115 0 0) 0x7fd2e8138180 con 0x7fd2e80b4d60 2024-03-25T18:21:20.238 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.235+0000 7fd2de7fc640 1 -- 172.21.15.97:0/850635344 --> v1:172.21.15.97:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd2e8142280 con 0x7fd2e80b4d60 2024-03-25T18:21:20.238 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.235+0000 7fd2de7fc640 1 -- 172.21.15.97:0/850635344 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7fd2d80018a0 con 0x7fd2e80b4d60 2024-03-25T18:21:20.238 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.235+0000 7fd2de7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:20.238 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.235+0000 7fd2de7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:21:20.238 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.235+0000 7fd2de7fc640 10 monclient(hunting): dump: 2024-03-25T18:21:20.238 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:21:20.238 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:21:20.238 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:21:20.238 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:21:20.238 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:20.238 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:21:20.239 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:21:20.239 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:21:20.239 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:21:20.239 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:21:20.254 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.252+0000 7fd2de7fc640 1 -- 172.21.15.97:0/850635344 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3028273459 0 0) 0x7fd2cc004eb0 con 0x7fd2e80b1730 2024-03-25T18:21:20.254 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.252+0000 7fd2de7fc640 1 -- 172.21.15.97:0/850635344 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd2e8138180 con 0x7fd2e80b1730 2024-03-25T18:21:20.254 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.252+0000 7fd2de7fc640 1 -- 172.21.15.97:0/850635344 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7fd2e4001c30 con 0x7fd2e80b1730 2024-03-25T18:21:20.254 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.252+0000 7fd2de7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:20.254 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.252+0000 7fd2de7fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:21:20.254 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.252+0000 7fd2de7fc640 10 monclient(hunting): dump: 2024-03-25T18:21:20.254 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:21:20.254 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:21:20.254 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:21:20.254 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:21:20.254 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:20.254 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:21:20.255 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:21:20.255 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:21:20.255 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:21:20.255 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:21:20.256 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.254+0000 7fd2de7fc640 1 -- 172.21.15.97:0/850635344 <== mon.0 v1:172.21.15.97:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2321729762 0 0) 0x7fd2e8142280 con 0x7fd2e80b4d60 2024-03-25T18:21:20.256 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.254+0000 7fd2de7fc640 1 -- 172.21.15.97:0/850635344 >> v1:172.21.15.97:6790/0 conn(0x7fd2e814be90 legacy=0x7fd2e81e2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:20.256 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.254+0000 7fd2de7fc640 1 -- 172.21.15.97:0/850635344 >> v1:172.21.15.98:6789/0 conn(0x7fd2e80b1730 legacy=0x7fd2e8131e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:20.256 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.254+0000 7fd2de7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:21:20.256 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.254+0000 7fd2de7fc640 1 monclient: found mon.a 2024-03-25T18:21:20.256 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.254+0000 7fd2de7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:21:20.256 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.254+0000 7fd2de7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:21:20.256 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.254+0000 7fd2de7fc640 1 -- 172.21.15.97:0/850635344 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2e8140e20 con 0x7fd2e80b4d60 2024-03-25T18:21:20.256 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.254+0000 7fd2de7fc640 10 monclient: _finish_auth 0 2024-03-25T18:21:20.256 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.254+0000 7fd2de7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:21:20.257 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.254+0000 7fd2de7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:21:20.257 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.254+0000 7fd2efaa1640 5 monclient: authenticate success, global_id 4449 2024-03-25T18:21:20.257 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.254+0000 7fd2efaa1640 10 monclient: _renew_subs 2024-03-25T18:21:20.257 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.254+0000 7fd2efaa1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:21:20.257 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.254+0000 7fd2efaa1640 1 -- 172.21.15.97:0/850635344 --> v1:172.21.15.97:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2e8132520 con 0x7fd2e80b4d60 2024-03-25T18:21:20.257 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.254+0000 7fd2efaa1640 10 client.4449.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:21:20.257 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.254+0000 7fd2efaa1640 10 monclient: _renew_subs 2024-03-25T18:21:20.257 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.254+0000 7fd2efaa1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:21:20.257 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.254+0000 7fd2efaa1640 1 -- 172.21.15.97:0/850635344 --> v1:172.21.15.97:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2e8145620 con 0x7fd2e80b4d60 2024-03-25T18:21:20.257 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.254+0000 7fd2efaa1640 1 librados: init done 2024-03-25T18:21:20.258 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.255+0000 7fd2efaa1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:20.258 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.255+0000 7fd2efaa1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:20.258 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.255+0000 7fd2efaa1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:21:20.258 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.255+0000 7fd2efaa1640 1 -- 172.21.15.97:0/850635344 --> v1:172.21.15.97:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2e8132820 con 0x7fd2e80b4d60 2024-03-25T18:21:20.521 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.520+0000 7fd2de7fc640 1 -- 172.21.15.97:0/850635344 <== mon.0 v1:172.21.15.97:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd2d8001d50 con 0x7fd2e80b4d60 2024-03-25T18:21:20.521 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.520+0000 7fd2de7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:21:20.521 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.520+0000 7fd2dffff640 4 set_mon_vals no callback set 2024-03-25T18:21:20.561 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.560+0000 7fd2de7fc640 1 -- 172.21.15.97:0/850635344 <== mon.0 v1:172.21.15.97:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7fd2d80049c0 con 0x7fd2e80b4d60 2024-03-25T18:21:20.561 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.560+0000 7fd2de7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:20.561 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.560+0000 7fd2de7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:21:20.562 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.560+0000 7fd2de7fc640 10 monclient: dump: 2024-03-25T18:21:20.562 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:21:20.562 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:21:20.562 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:21:20.562 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:21:20.562 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:20.562 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:21:20.562 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:21:20.562 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:21:20.562 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:21:20.562 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:21:20.563 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.561+0000 7fd2de7fc640 1 -- 172.21.15.97:0/850635344 <== mon.0 v1:172.21.15.97:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2377622137 0 0) 0x7fd2d801a9d0 con 0x7fd2e80b4d60 2024-03-25T18:21:20.563 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.561+0000 7fd2ed83d640 10 monclient: get_auth_request con 0x7fd2cc06e070 auth_method 0 2024-03-25T18:21:20.563 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.562+0000 7fd2de7fc640 10 client.4449.objecter ms_handle_connect 0x7fd2cc06e070 2024-03-25T18:21:20.569 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.567+0000 7fd2de7fc640 1 -- 172.21.15.97:0/850635344 <== mon.0 v1:172.21.15.97:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 430711918 0 0) 0x7fd2d8054710 con 0x7fd2e80b4d60 2024-03-25T18:21:20.569 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.567+0000 7fd2de7fc640 10 client.4449.objecter ms_dispatch 0x7fd2e8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:21:20.569 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.567+0000 7fd2de7fc640 3 client.4449.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:21:20.569 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.567+0000 7fd2de7fc640 3 client.4449.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:21:20.569 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.568+0000 7fd2de7fc640 20 client.4449.objecter dump_active .. 0 homeless 2024-03-25T18:21:20.569 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.568+0000 7fd2de7fc640 1 -- 172.21.15.97:0/850635344 <== mon.0 v1:172.21.15.97:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd2d8050050 con 0x7fd2e80b4d60 2024-03-25T18:21:20.569 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.568+0000 7fd2de7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:20.569 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.568+0000 7fd2de7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:21:20.814 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.812+0000 7fd2efaa1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:20.814 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.812+0000 7fd2efaa1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:20.814 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.812+0000 7fd2efaa1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:21:20.814 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.812+0000 7fd2efaa1640 1 -- 172.21.15.97:0/850635344 --> v1:172.21.15.97:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd2e80627b0 con 0x7fd2e80b4d60 2024-03-25T18:21:20.867 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.866+0000 7fd2de7fc640 1 -- 172.21.15.97:0/850635344 <== mon.0 v1:172.21.15.97:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2169427412) 0x7fd2d8052150 con 0x7fd2e80b4d60 2024-03-25T18:21:20.867 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.866+0000 7fd2de7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:20.867 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.866+0000 7fd2de7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:21:20.867 INFO:teuthology.misc.health.smithi097.stdout: 2024-03-25T18:21:20.867 INFO:teuthology.misc.health.smithi097.stdout:{"epoch":10,"fsid":"4c3d472c-79b8-486b-9e3d-2738d4306754","created":"2024-03-25T18:16:14.737309+0000","modified":"2024-03-25T18:16:20.357492+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:20.357492+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cb5152b9-262f-45c3-94b8-18d0e81df558","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6daf8793-0838-473d-a7d6-9a0d2894c75d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"136f1493-3b9c-4f43-a2e7-641482d69dae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"397d538c-4f64-4daf-9e8b-662d09f91327","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4f1df08-663f-4bda-93f8-3878ab7c24b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c4b18d70-a3ce-4b15-9b70-eae5525bd761","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8dd4033a-de0e-4f8a-96dc-59f3c1f825b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0077722e-ffa6-4a3d-9891-929d9869b286","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:21:20.870 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.868+0000 7fd2efaa1640 10 librados: watch_flush enter 2024-03-25T18:21:20.870 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.868+0000 7fd2efaa1640 10 librados: watch_flush exit 2024-03-25T18:21:20.870 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.868+0000 7fd2efaa1640 20 client.4449.objecter shutdown clearing up homeless session... 2024-03-25T18:21:20.870 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.868+0000 7fd2efaa1640 10 client.4449.objecter successfully canceled tick 2024-03-25T18:21:20.870 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.868+0000 7fd2efaa1640 1 -- 172.21.15.97:0/850635344 >> v1:172.21.15.98:6800/800387573 conn(0x7fd2cc06e070 legacy=0x7fd2cc070530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:20.870 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.868+0000 7fd2efaa1640 10 monclient: shutdown 2024-03-25T18:21:20.870 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.868+0000 7fd2efaa1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:21:20.870 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.868+0000 7fd2efaa1640 1 -- 172.21.15.97:0/850635344 >> v1:172.21.15.97:6789/0 conn(0x7fd2e80b4d60 legacy=0x7fd2e81d7590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:20.870 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.868+0000 7fd2efaa1640 1 -- 172.21.15.97:0/850635344 shutdown_connections 2024-03-25T18:21:20.871 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.868+0000 7fd2efaa1640 1 -- 172.21.15.97:0/850635344 >> 172.21.15.97:0/850635344 conn(0x7fd2e80617a0 msgr2=0x7fd2e8061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:21:20.871 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.868+0000 7fd2efaa1640 1 -- 172.21.15.97:0/850635344 shutdown_connections 2024-03-25T18:21:20.871 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.868+0000 7fd2efaa1640 1 -- 172.21.15.97:0/850635344 wait complete. 2024-03-25T18:21:20.871 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:20.868+0000 7fd2efaa1640 1 librados: shutdown 2024-03-25T18:21:20.880 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:21:27.687 DEBUG:teuthology.orchestra.run.smithi097:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:21:27.804 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:27.803+0000 7f9fde633640 10 monclient: get_monmap_and_config 2024-03-25T18:21:27.804 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:27.803+0000 7f9fde633640 10 monclient: build_initial_monmap 2024-03-25T18:21:27.804 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:27.803+0000 7f9fde633640 10 monclient: monmap: 2024-03-25T18:21:27.804 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:21:27.804 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:21:27.804 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:21:27.804156+0000 2024-03-25T18:21:27.804 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:21:27.804156+0000 2024-03-25T18:21:27.805 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:21:27.805 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:21:27.805 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:21:27.805 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T18:21:27.805 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-b 2024-03-25T18:21:27.805 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:21:27.805 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:27.805+0000 7f9fde633640 1 Processor -- start 2024-03-25T18:21:27.805 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:27.805+0000 7f9fde633640 1 -- start start 2024-03-25T18:21:27.806 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:27.805+0000 7f9fde633640 10 monclient: init 2024-03-25T18:21:27.806 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:27.805+0000 7f9fde633640 10 monclient: _reopen_session rank -1 2024-03-25T18:21:27.806 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:27.805+0000 7f9fde633640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:21:27.806 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:27.805+0000 7f9fde633640 10 monclient(hunting): picked mon.noname-a con 0x7f9fd80a98f0 addr v1:172.21.15.97:6789/0 2024-03-25T18:21:27.806 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:27.805+0000 7f9fde633640 10 monclient(hunting): picked mon.noname-b con 0x7f9fd813f710 addr v1:172.21.15.97:6790/0 2024-03-25T18:21:27.806 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:27.806+0000 7f9fde633640 10 monclient(hunting): picked mon.noname-c con 0x7f9fd81430c0 addr v1:172.21.15.98:6789/0 2024-03-25T18:21:27.806 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:27.806+0000 7f9fde633640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fd8142520 con 0x7f9fd80a98f0 2024-03-25T18:21:27.807 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:27.806+0000 7f9fde633640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fd80b28a0 con 0x7f9fd81430c0 2024-03-25T18:21:27.807 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:27.806+0000 7f9fde633640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fd80b2a30 con 0x7f9fd813f710 2024-03-25T18:21:27.807 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:27.806+0000 7f9fde633640 10 monclient(hunting): _renew_subs 2024-03-25T18:21:27.807 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:27.806+0000 7f9fde633640 10 monclient(hunting): authenticate will time out at 7900.080923s 2024-03-25T18:21:27.822 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:27.822+0000 7f9fdcbd0640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f9fd81430c0 0x7f9fd814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.97:54596/0 (socket says 172.21.15.97:54596) 2024-03-25T18:21:27.822 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:27.822+0000 7f9fdcbd0640 1 -- 172.21.15.97:0/3046952587 learned_addr learned my addr 172.21.15.97:0/3046952587 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:21:27.833 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:27.832+0000 7f9fd77fe640 1 -- 172.21.15.97:0/3046952587 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1710171361 0 0) 0x7f9fd80b28a0 con 0x7f9fd81430c0 2024-03-25T18:21:27.833 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:27.832+0000 7f9fd77fe640 10 monclient(hunting): _negotiate 2024-03-25T18:21:27.833 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:27.832+0000 7f9fd77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:27.833 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:27.832+0000 7f9fd77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:27.833 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:27.832+0000 7f9fd77fe640 10 monclient(hunting): my global_id is 4453 2024-03-25T18:21:27.833 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:27.832+0000 7f9fd77fe640 1 -- 172.21.15.97:0/3046952587 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9fc4003520 con 0x7f9fd81430c0 2024-03-25T18:21:27.844 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:27.843+0000 7f9fd77fe640 1 -- 172.21.15.97:0/3046952587 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 269150592 0 0) 0x7f9fd8142520 con 0x7f9fd80a98f0 2024-03-25T18:21:27.844 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:27.843+0000 7f9fd77fe640 10 monclient(hunting): _negotiate 2024-03-25T18:21:27.844 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:27.843+0000 7f9fd77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:27.844 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:27.843+0000 7f9fd77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:27.844 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:27.843+0000 7f9fd77fe640 10 monclient(hunting): my global_id is 4452 2024-03-25T18:21:27.844 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:27.843+0000 7f9fd77fe640 1 -- 172.21.15.97:0/3046952587 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9fd80b28a0 con 0x7f9fd80a98f0 2024-03-25T18:21:27.848 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:27.847+0000 7f9fd77fe640 1 -- 172.21.15.97:0/3046952587 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4110030129 0 0) 0x7f9fc4003520 con 0x7f9fd81430c0 2024-03-25T18:21:27.848 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:27.847+0000 7f9fd77fe640 1 -- 172.21.15.97:0/3046952587 >> v1:172.21.15.97:6790/0 conn(0x7f9fd813f710 legacy=0x7f9fd81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:27.848 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:27.847+0000 7f9fd77fe640 1 -- 172.21.15.97:0/3046952587 >> v1:172.21.15.97:6789/0 conn(0x7f9fd80a98f0 legacy=0x7f9fd80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:27.848 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:27.847+0000 7f9fd77fe640 10 monclient: _finish_hunting 0 2024-03-25T18:21:27.848 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:27.847+0000 7f9fd77fe640 1 monclient: found mon.noname-c 2024-03-25T18:21:27.848 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:27.847+0000 7f9fd77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:21:27.848 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:27.847+0000 7f9fd77fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.98:6789/0 2024-03-25T18:21:27.848 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:27.847+0000 7f9fd77fe640 1 -- 172.21.15.97:0/3046952587 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fd80b44c0 con 0x7f9fd81430c0 2024-03-25T18:21:27.848 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:27.847+0000 7f9fd77fe640 10 monclient: _finish_auth 0 2024-03-25T18:21:27.848 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:27.847+0000 7f9fd77fe640 10 monclient: _check_auth_tickets 2024-03-25T18:21:27.848 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:27.847+0000 7f9fd77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:21:27.849 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:27.847+0000 7f9fd77fe640 1 -- 172.21.15.97:0/3046952587 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f9fd0001a10 con 0x7f9fd81430c0 2024-03-25T18:21:27.849 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:27.847+0000 7f9fde633640 5 monclient: authenticate success, global_id 4453 2024-03-25T18:21:27.849 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:27.847+0000 7f9fde633640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:21:27.849 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:27.847+0000 7f9fd77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:27.849 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:27.847+0000 7f9fd77fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:21:27.849 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:27.847+0000 7f9fd77fe640 10 monclient: dump: 2024-03-25T18:21:27.849 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:21:27.849 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:21:27.849 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:21:27.849 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:21:27.849 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:27.849 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:21:27.849 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:21:27.849 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:21:27.849 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:21:27.850 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:21:27.850 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:27.847+0000 7f9fde633640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:21:27.856 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:27.856+0000 7f9fd77fe640 1 -- 172.21.15.97:0/3046952587 <== mon.1 v1:172.21.15.98:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9fd0001b70 con 0x7f9fd81430c0 2024-03-25T18:21:27.856 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:27.856+0000 7f9fd77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:21:27.856 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:27.856+0000 7f9fd77fe640 1 -- 172.21.15.97:0/3046952587 <== mon.1 v1:172.21.15.98:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f9fd0001f80 con 0x7f9fd81430c0 2024-03-25T18:21:27.857 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:27.856+0000 7f9fde633640 10 monclient: get_monmap_and_config success 2024-03-25T18:21:27.857 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:27.856+0000 7f9fde633640 4 set_mon_vals no callback set 2024-03-25T18:21:27.857 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:27.856+0000 7f9fd77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:27.857 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:27.856+0000 7f9fd77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:21:27.857 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:27.856+0000 7f9fd77fe640 10 monclient: dump: 2024-03-25T18:21:27.857 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:21:27.857 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:21:27.857 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:21:27.857 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:21:27.857 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:27.857 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:21:27.857 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:21:27.857 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:21:27.857 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:21:27.857 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:21:27.859 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:27.857+0000 7f9fde633640 10 monclient: shutdown 2024-03-25T18:21:27.859 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:27.857+0000 7f9fde633640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:21:27.860 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:27.857+0000 7f9fde633640 1 -- 172.21.15.97:0/3046952587 >> v1:172.21.15.98:6789/0 conn(0x7f9fd81430c0 legacy=0x7f9fd814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:27.860 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:27.857+0000 7f9fde633640 1 -- 172.21.15.97:0/3046952587 shutdown_connections 2024-03-25T18:21:27.860 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:27.857+0000 7f9fde633640 1 -- 172.21.15.97:0/3046952587 >> 172.21.15.97:0/3046952587 conn(0x7f9fd80b7ea0 msgr2=0x7f9fd80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:21:27.860 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:27.857+0000 7f9fde633640 1 -- 172.21.15.97:0/3046952587 shutdown_connections 2024-03-25T18:21:27.860 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:27.857+0000 7f9fde633640 1 -- 172.21.15.97:0/3046952587 wait complete. 2024-03-25T18:21:27.860 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:27.858+0000 7f9fde633640 10 monclient: build_initial_monmap 2024-03-25T18:21:27.860 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:27.858+0000 7f9fde633640 10 monclient: monmap: 2024-03-25T18:21:27.860 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:21:27.860 INFO:teuthology.misc.health.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:21:27.860 INFO:teuthology.misc.health.smithi097.stderr:last_changed 0.000000 2024-03-25T18:21:27.860 INFO:teuthology.misc.health.smithi097.stderr:created 0.000000 2024-03-25T18:21:27.860 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:21:27.860 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 1 2024-03-25T18:21:27.860 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:21:27.860 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T18:21:27.861 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-c 2024-03-25T18:21:27.861 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:21:27.861 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:27.858+0000 7f9fde633640 1 librados: starting msgr at 2024-03-25T18:21:27.861 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:27.858+0000 7f9fde633640 1 librados: starting objecter 2024-03-25T18:21:27.861 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:27.858+0000 7f9fde633640 1 Processor -- start 2024-03-25T18:21:27.861 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:27.858+0000 7f9fde633640 1 -- start start 2024-03-25T18:21:27.861 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:27.858+0000 7f9fde633640 1 librados: setting wanted keys 2024-03-25T18:21:27.861 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:27.858+0000 7f9fde633640 1 librados: calling monclient init 2024-03-25T18:21:27.861 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:27.858+0000 7f9fde633640 10 monclient: init 2024-03-25T18:21:27.861 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:27.859+0000 7f9fde633640 10 monclient: _reopen_session rank -1 2024-03-25T18:21:27.861 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:27.859+0000 7f9fde633640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:21:27.861 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:27.859+0000 7f9fde633640 10 monclient(hunting): picked mon.noname-b con 0x7f9fd80a98f0 addr v1:172.21.15.98:6789/0 2024-03-25T18:21:27.861 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:27.859+0000 7f9fde633640 10 monclient(hunting): picked mon.noname-a con 0x7f9fd813f710 addr v1:172.21.15.97:6789/0 2024-03-25T18:21:27.861 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:27.859+0000 7f9fde633640 10 monclient(hunting): picked mon.noname-c con 0x7f9fd81430c0 addr v1:172.21.15.97:6790/0 2024-03-25T18:21:27.861 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:27.859+0000 7f9fde633640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fd813c2b0 con 0x7f9fd813f710 2024-03-25T18:21:27.862 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:27.859+0000 7f9fde633640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fd8136f10 con 0x7f9fd80a98f0 2024-03-25T18:21:27.862 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:27.859+0000 7f9fde633640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fd81370a0 con 0x7f9fd81430c0 2024-03-25T18:21:27.862 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:27.859+0000 7f9fde633640 10 monclient(hunting): _renew_subs 2024-03-25T18:21:27.862 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:27.859+0000 7f9fde633640 10 monclient(hunting): authenticate will time out at 7900.134258s 2024-03-25T18:21:27.872 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:27.872+0000 7f9fd7fff640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f9fd80a98f0 0x7f9fd81d7590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.97:54650/0 (socket says 172.21.15.97:54650) 2024-03-25T18:21:27.873 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:27.872+0000 7f9fd7fff640 1 -- 172.21.15.97:0/2508867422 learned_addr learned my addr 172.21.15.97:0/2508867422 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:21:27.873 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:27.873+0000 7f9fd57fa640 10 client.?.objecter ms_handle_connect 0x7f9fd80a98f0 2024-03-25T18:21:27.873 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:27.873+0000 7f9fd57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:27.880 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:27.880+0000 7f9fd57fa640 10 client.?.objecter ms_handle_connect 0x7f9fd81430c0 2024-03-25T18:21:27.880 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:27.880+0000 7f9fd57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:27.880 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:27.880+0000 7f9fd57fa640 1 -- 172.21.15.97:0/2508867422 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3090466227 0 0) 0x7f9fd8136f10 con 0x7f9fd80a98f0 2024-03-25T18:21:27.880 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:27.880+0000 7f9fd57fa640 10 monclient(hunting): _negotiate 2024-03-25T18:21:27.880 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:27.880+0000 7f9fd57fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:27.881 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:27.880+0000 7f9fd57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:27.881 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:27.880+0000 7f9fd57fa640 10 monclient(hunting): my global_id is 4456 2024-03-25T18:21:27.881 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:27.880+0000 7f9fd57fa640 1 -- 172.21.15.97:0/2508867422 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9fb8004eb0 con 0x7f9fd80a98f0 2024-03-25T18:21:27.887 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:27.886+0000 7f9fd57fa640 10 client.?.objecter ms_handle_connect 0x7f9fd813f710 2024-03-25T18:21:27.887 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:27.886+0000 7f9fd57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:27.893 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:27.892+0000 7f9fd57fa640 1 -- 172.21.15.97:0/2508867422 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 79375288 0 0) 0x7f9fb8004eb0 con 0x7f9fd80a98f0 2024-03-25T18:21:27.893 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:27.892+0000 7f9fd57fa640 1 -- 172.21.15.97:0/2508867422 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9fd8136f10 con 0x7f9fd80a98f0 2024-03-25T18:21:27.893 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:27.893+0000 7f9fd57fa640 1 -- 172.21.15.97:0/2508867422 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f9fc8001a10 con 0x7f9fd80a98f0 2024-03-25T18:21:27.893 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:27.893+0000 7f9fd57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:27.893 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:27.893+0000 7f9fd57fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:21:27.893 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:27.893+0000 7f9fd57fa640 10 monclient(hunting): dump: 2024-03-25T18:21:27.893 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:21:27.893 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:21:27.893 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:21:27.894 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:21:27.894 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:27.894 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:21:27.894 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:21:27.894 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:21:27.894 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:21:27.894 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:21:27.894 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:27.893+0000 7f9fd57fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:21:27.897 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:27.897+0000 7f9fd57fa640 1 -- 172.21.15.97:0/2508867422 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 327020993 0 0) 0x7f9fd81370a0 con 0x7f9fd81430c0 2024-03-25T18:21:27.898 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:27.897+0000 7f9fd57fa640 10 monclient(hunting): _negotiate 2024-03-25T18:21:27.898 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:27.897+0000 7f9fd57fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:27.898 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:27.897+0000 7f9fd57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:27.898 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:27.897+0000 7f9fd57fa640 10 monclient(hunting): my global_id is 4448 2024-03-25T18:21:27.898 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:27.897+0000 7f9fd57fa640 1 -- 172.21.15.97:0/2508867422 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9fb8004eb0 con 0x7f9fd81430c0 2024-03-25T18:21:27.906 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:27.906+0000 7f9fd57fa640 1 -- 172.21.15.97:0/2508867422 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3986176395 0 0) 0x7f9fd813c2b0 con 0x7f9fd813f710 2024-03-25T18:21:27.906 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:27.906+0000 7f9fd57fa640 10 monclient(hunting): _negotiate 2024-03-25T18:21:27.906 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:27.906+0000 7f9fd57fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:27.906 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:27.906+0000 7f9fd57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:27.907 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:27.906+0000 7f9fd57fa640 10 monclient(hunting): my global_id is 4455 2024-03-25T18:21:27.907 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:27.906+0000 7f9fd57fa640 1 -- 172.21.15.97:0/2508867422 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9fd81370a0 con 0x7f9fd813f710 2024-03-25T18:21:27.916 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:27.915+0000 7f9fd57fa640 1 -- 172.21.15.97:0/2508867422 <== mon.1 v1:172.21.15.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3218403622 0 0) 0x7f9fd8136f10 con 0x7f9fd80a98f0 2024-03-25T18:21:27.916 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:27.915+0000 7f9fd57fa640 1 -- 172.21.15.97:0/2508867422 >> v1:172.21.15.97:6790/0 conn(0x7f9fd81430c0 legacy=0x7f9fd8133000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:27.916 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:27.915+0000 7f9fd57fa640 1 -- 172.21.15.97:0/2508867422 >> v1:172.21.15.97:6789/0 conn(0x7f9fd813f710 legacy=0x7f9fd81377d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:27.916 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:27.915+0000 7f9fd57fa640 10 monclient: _finish_hunting 0 2024-03-25T18:21:27.916 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:27.915+0000 7f9fd57fa640 1 monclient: found mon.b 2024-03-25T18:21:27.916 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:27.915+0000 7f9fd57fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:21:27.916 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:27.915+0000 7f9fd57fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:21:27.916 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:27.915+0000 7f9fd57fa640 1 -- 172.21.15.97:0/2508867422 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fd8137230 con 0x7f9fd80a98f0 2024-03-25T18:21:27.916 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:27.915+0000 7f9fd57fa640 10 monclient: _finish_auth 0 2024-03-25T18:21:27.916 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:27.915+0000 7f9fd57fa640 10 monclient: _check_auth_tickets 2024-03-25T18:21:27.917 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:27.915+0000 7f9fd57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:21:27.917 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:27.915+0000 7f9fde633640 5 monclient: authenticate success, global_id 4456 2024-03-25T18:21:27.917 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:27.915+0000 7f9fde633640 10 monclient: _renew_subs 2024-03-25T18:21:27.917 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:27.915+0000 7f9fde633640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:21:27.917 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:27.915+0000 7f9fde633640 1 -- 172.21.15.97:0/2508867422 --> v1:172.21.15.98:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9fd813b620 con 0x7f9fd80a98f0 2024-03-25T18:21:27.917 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:27.915+0000 7f9fde633640 10 client.4456.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:21:27.917 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:27.915+0000 7f9fde633640 10 monclient: _renew_subs 2024-03-25T18:21:27.917 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:27.915+0000 7f9fde633640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:21:27.917 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:27.915+0000 7f9fde633640 1 -- 172.21.15.97:0/2508867422 --> v1:172.21.15.98:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fd8133710 con 0x7f9fd80a98f0 2024-03-25T18:21:27.917 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:27.915+0000 7f9fde633640 1 librados: init done 2024-03-25T18:21:27.918 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:27.916+0000 7f9fde633640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:27.918 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:27.916+0000 7f9fde633640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:27.918 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:27.916+0000 7f9fde633640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:21:27.918 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:27.916+0000 7f9fde633640 1 -- 172.21.15.97:0/2508867422 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9fd80627b0 con 0x7f9fd80a98f0 2024-03-25T18:21:28.188 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:28.187+0000 7f9fd57fa640 1 -- 172.21.15.97:0/2508867422 <== mon.1 v1:172.21.15.98:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9fc8001c90 con 0x7f9fd80a98f0 2024-03-25T18:21:28.188 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:28.187+0000 7f9fd57fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:21:28.188 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:28.187+0000 7f9fd57fa640 1 -- 172.21.15.97:0/2508867422 <== mon.1 v1:172.21.15.98:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f9fc80038c0 con 0x7f9fd80a98f0 2024-03-25T18:21:28.188 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:28.187+0000 7f9fd57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:28.188 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:28.187+0000 7f9fd57fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:21:28.188 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:28.187+0000 7f9fd57fa640 10 monclient: dump: 2024-03-25T18:21:28.188 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:21:28.188 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:21:28.188 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:21:28.188 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:21:28.188 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:28.189 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:21:28.189 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:21:28.189 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:21:28.189 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:21:28.189 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:21:28.189 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:28.187+0000 7f9fd6ffd640 4 set_mon_vals no callback set 2024-03-25T18:21:28.189 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:28.188+0000 7f9fd57fa640 1 -- 172.21.15.97:0/2508867422 <== mon.1 v1:172.21.15.98:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2377622137 0 0) 0x7f9fc8003ae0 con 0x7f9fd80a98f0 2024-03-25T18:21:28.190 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:28.189+0000 7f9fcffff640 10 monclient: get_auth_request con 0x7f9fb806de80 auth_method 0 2024-03-25T18:21:28.190 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:28.189+0000 7f9fd57fa640 10 client.4456.objecter ms_handle_connect 0x7f9fb806de80 2024-03-25T18:21:28.194 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:28.194+0000 7f9fd57fa640 1 -- 172.21.15.97:0/2508867422 <== mon.1 v1:172.21.15.98:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 430711918 0 0) 0x7f9fc80546b0 con 0x7f9fd80a98f0 2024-03-25T18:21:28.194 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:28.194+0000 7f9fd57fa640 10 client.4456.objecter ms_dispatch 0x7f9fd8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:21:28.194 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:28.194+0000 7f9fd57fa640 3 client.4456.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:21:28.195 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:28.194+0000 7f9fd57fa640 3 client.4456.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:21:28.195 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:28.194+0000 7f9fd57fa640 20 client.4456.objecter dump_active .. 0 homeless 2024-03-25T18:21:28.195 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:28.194+0000 7f9fd57fa640 1 -- 172.21.15.97:0/2508867422 <== mon.1 v1:172.21.15.98:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9fc80501c0 con 0x7f9fd80a98f0 2024-03-25T18:21:28.195 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:28.194+0000 7f9fd57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:28.195 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:28.194+0000 7f9fd57fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:21:28.429 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:28.428+0000 7f9fde633640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:28.429 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:28.428+0000 7f9fde633640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:28.429 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:28.428+0000 7f9fde633640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:21:28.429 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:28.428+0000 7f9fde633640 1 -- 172.21.15.97:0/2508867422 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9fd81373f0 con 0x7f9fd80a98f0 2024-03-25T18:21:28.468 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:28.468+0000 7f9fd57fa640 1 -- 172.21.15.97:0/2508867422 <== mon.1 v1:172.21.15.98:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2169427412) 0x7f9fc80520f0 con 0x7f9fd80a98f0 2024-03-25T18:21:28.468 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:28.468+0000 7f9fd57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:28.468 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:28.468+0000 7f9fd57fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:21:28.469 INFO:teuthology.misc.health.smithi097.stdout: 2024-03-25T18:21:28.469 INFO:teuthology.misc.health.smithi097.stdout:{"epoch":10,"fsid":"4c3d472c-79b8-486b-9e3d-2738d4306754","created":"2024-03-25T18:16:14.737309+0000","modified":"2024-03-25T18:16:20.357492+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:20.357492+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cb5152b9-262f-45c3-94b8-18d0e81df558","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6daf8793-0838-473d-a7d6-9a0d2894c75d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"136f1493-3b9c-4f43-a2e7-641482d69dae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"397d538c-4f64-4daf-9e8b-662d09f91327","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4f1df08-663f-4bda-93f8-3878ab7c24b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c4b18d70-a3ce-4b15-9b70-eae5525bd761","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8dd4033a-de0e-4f8a-96dc-59f3c1f825b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0077722e-ffa6-4a3d-9891-929d9869b286","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:21:28.471 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:28.470+0000 7f9fde633640 10 librados: watch_flush enter 2024-03-25T18:21:28.471 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:28.470+0000 7f9fde633640 10 librados: watch_flush exit 2024-03-25T18:21:28.471 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:28.470+0000 7f9fde633640 20 client.4456.objecter shutdown clearing up homeless session... 2024-03-25T18:21:28.471 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:28.470+0000 7f9fde633640 10 client.4456.objecter successfully canceled tick 2024-03-25T18:21:28.472 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:28.470+0000 7f9fde633640 1 -- 172.21.15.97:0/2508867422 >> v1:172.21.15.98:6800/800387573 conn(0x7f9fb806de80 legacy=0x7f9fb8070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:28.472 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:28.470+0000 7f9fde633640 10 monclient: shutdown 2024-03-25T18:21:28.472 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:28.470+0000 7f9fde633640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:21:28.472 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:28.470+0000 7f9fde633640 1 -- 172.21.15.97:0/2508867422 >> v1:172.21.15.98:6789/0 conn(0x7f9fd80a98f0 legacy=0x7f9fd81d7590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:28.472 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:28.470+0000 7f9fde633640 1 -- 172.21.15.97:0/2508867422 shutdown_connections 2024-03-25T18:21:28.472 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:28.470+0000 7f9fde633640 1 -- 172.21.15.97:0/2508867422 >> 172.21.15.97:0/2508867422 conn(0x7f9fd80b7ea0 msgr2=0x7f9fd8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:21:28.472 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:28.471+0000 7f9fde633640 1 -- 172.21.15.97:0/2508867422 shutdown_connections 2024-03-25T18:21:28.472 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:28.471+0000 7f9fde633640 1 -- 172.21.15.97:0/2508867422 wait complete. 2024-03-25T18:21:28.472 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:28.471+0000 7f9fde633640 1 librados: shutdown 2024-03-25T18:21:28.483 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:21:32.875 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:05:34.914 34931== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:32.920 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:05:34.959 34931== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:32.924 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:05:34.967 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:32.926 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:05:34.969 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:32.942 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:05:34.986 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:32.966 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:05:35.005 34931== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:33.007 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:05:35.045 34935== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:33.024 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:05:35.062 34935== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:33.055 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:05:35.094 34935== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:35.289 DEBUG:teuthology.orchestra.run.smithi097:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:21:35.408 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.407+0000 7f76e9582640 10 monclient: get_monmap_and_config 2024-03-25T18:21:35.408 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.407+0000 7f76e9582640 10 monclient: build_initial_monmap 2024-03-25T18:21:35.408 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.407+0000 7f76e9582640 10 monclient: monmap: 2024-03-25T18:21:35.408 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:21:35.408 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:21:35.408 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:21:35.408070+0000 2024-03-25T18:21:35.408 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:21:35.408070+0000 2024-03-25T18:21:35.408 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:21:35.408 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:21:35.408 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:21:35.408 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T18:21:35.409 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-b 2024-03-25T18:21:35.409 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:21:35.410 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.408+0000 7f76e9582640 1 Processor -- start 2024-03-25T18:21:35.410 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.408+0000 7f76e9582640 1 -- start start 2024-03-25T18:21:35.410 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.408+0000 7f76e9582640 10 monclient: init 2024-03-25T18:21:35.410 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.409+0000 7f76e9582640 10 monclient: _reopen_session rank -1 2024-03-25T18:21:35.410 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.409+0000 7f76e9582640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:21:35.410 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.409+0000 7f76e9582640 10 monclient(hunting): picked mon.noname-a con 0x7f76e40a98f0 addr v1:172.21.15.97:6789/0 2024-03-25T18:21:35.410 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.409+0000 7f76e9582640 10 monclient(hunting): picked mon.noname-b con 0x7f76e413f710 addr v1:172.21.15.97:6790/0 2024-03-25T18:21:35.410 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.409+0000 7f76e9582640 10 monclient(hunting): picked mon.noname-c con 0x7f76e41430c0 addr v1:172.21.15.98:6789/0 2024-03-25T18:21:35.410 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.409+0000 7f76e9582640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76e4142520 con 0x7f76e40a98f0 2024-03-25T18:21:35.410 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.409+0000 7f76e9582640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76e40b28a0 con 0x7f76e41430c0 2024-03-25T18:21:35.410 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.409+0000 7f76e9582640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76e40b2a30 con 0x7f76e413f710 2024-03-25T18:21:35.410 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.409+0000 7f76e9582640 10 monclient(hunting): _renew_subs 2024-03-25T18:21:35.411 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.409+0000 7f76e9582640 10 monclient(hunting): authenticate will time out at 7907.684436s 2024-03-25T18:21:35.421 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.421+0000 7f76e2ffd640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f76e40a98f0 0x7f76e40a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:41688/0 (socket says 172.21.15.97:41688) 2024-03-25T18:21:35.421 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.421+0000 7f76e2ffd640 1 -- 172.21.15.97:0/1914405770 learned_addr learned my addr 172.21.15.97:0/1914405770 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:21:35.434 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.433+0000 7f76e1ffb640 1 -- 172.21.15.97:0/1914405770 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1821143831 0 0) 0x7f76e40b28a0 con 0x7f76e41430c0 2024-03-25T18:21:35.434 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.433+0000 7f76e1ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:21:35.434 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.433+0000 7f76e1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:35.434 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.433+0000 7f76e1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:35.434 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.433+0000 7f76e1ffb640 10 monclient(hunting): my global_id is 4459 2024-03-25T18:21:35.434 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.433+0000 7f76e1ffb640 1 -- 172.21.15.97:0/1914405770 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76cc003520 con 0x7f76e41430c0 2024-03-25T18:21:35.434 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.434+0000 7f76e1ffb640 1 -- 172.21.15.97:0/1914405770 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2521271068 0 0) 0x7f76e40b2a30 con 0x7f76e413f710 2024-03-25T18:21:35.434 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.434+0000 7f76e1ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:21:35.435 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.434+0000 7f76e1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:35.435 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.434+0000 7f76e1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:35.435 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.434+0000 7f76e1ffb640 10 monclient(hunting): my global_id is 4451 2024-03-25T18:21:35.435 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.434+0000 7f76e1ffb640 1 -- 172.21.15.97:0/1914405770 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76e40b28a0 con 0x7f76e413f710 2024-03-25T18:21:35.437 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.437+0000 7f76e1ffb640 1 -- 172.21.15.97:0/1914405770 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 270509758 0 0) 0x7f76e4142520 con 0x7f76e40a98f0 2024-03-25T18:21:35.437 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.437+0000 7f76e1ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:21:35.438 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.437+0000 7f76e1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:35.438 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.437+0000 7f76e1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:35.438 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.437+0000 7f76e1ffb640 10 monclient(hunting): my global_id is 4458 2024-03-25T18:21:35.438 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.437+0000 7f76e1ffb640 1 -- 172.21.15.97:0/1914405770 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76e40b2a30 con 0x7f76e40a98f0 2024-03-25T18:21:35.444 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.444+0000 7f76e1ffb640 1 -- 172.21.15.97:0/1914405770 <== mon.2 v1:172.21.15.97:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 49177414 0 0) 0x7f76e40b28a0 con 0x7f76e413f710 2024-03-25T18:21:35.445 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.444+0000 7f76e1ffb640 1 -- 172.21.15.97:0/1914405770 >> v1:172.21.15.98:6789/0 conn(0x7f76e41430c0 legacy=0x7f76e414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:35.445 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.444+0000 7f76e1ffb640 1 -- 172.21.15.97:0/1914405770 >> v1:172.21.15.97:6789/0 conn(0x7f76e40a98f0 legacy=0x7f76e40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:35.445 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.444+0000 7f76e1ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:21:35.445 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.444+0000 7f76e1ffb640 1 monclient: found mon.noname-b 2024-03-25T18:21:35.445 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.444+0000 7f76e1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:21:35.445 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.444+0000 7f76e1ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.97:6790/0 2024-03-25T18:21:35.445 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.444+0000 7f76e1ffb640 1 -- 172.21.15.97:0/1914405770 --> v1:172.21.15.97:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76e40b44c0 con 0x7f76e413f710 2024-03-25T18:21:35.445 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.444+0000 7f76e1ffb640 10 monclient: _finish_auth 0 2024-03-25T18:21:35.445 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.444+0000 7f76e1ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:21:35.445 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.444+0000 7f76e1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:21:35.445 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.444+0000 7f76e1ffb640 1 -- 172.21.15.97:0/1914405770 <== mon.2 v1:172.21.15.97:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f76d8001670 con 0x7f76e413f710 2024-03-25T18:21:35.445 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.444+0000 7f76e1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:35.445 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.444+0000 7f76e1ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:21:35.445 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.444+0000 7f76e1ffb640 10 monclient: dump: 2024-03-25T18:21:35.446 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:21:35.446 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:21:35.446 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:21:35.446 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:21:35.446 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:35.446 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:21:35.446 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:21:35.446 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:21:35.446 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:21:35.446 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:21:35.446 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.444+0000 7f76e9582640 5 monclient: authenticate success, global_id 4451 2024-03-25T18:21:35.446 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.444+0000 7f76e9582640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:21:35.460 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.459+0000 7f76e1ffb640 1 -- 172.21.15.97:0/1914405770 <== mon.2 v1:172.21.15.97:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f76d80017d0 con 0x7f76e413f710 2024-03-25T18:21:35.460 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.459+0000 7f76e1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:21:35.460 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.459+0000 7f76e1ffb640 1 -- 172.21.15.97:0/1914405770 <== mon.2 v1:172.21.15.97:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f76d8001be0 con 0x7f76e413f710 2024-03-25T18:21:35.460 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.459+0000 7f76e9582640 10 monclient: get_monmap_and_config success 2024-03-25T18:21:35.460 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.459+0000 7f76e9582640 4 set_mon_vals no callback set 2024-03-25T18:21:35.460 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.459+0000 7f76e1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:35.460 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.459+0000 7f76e1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:21:35.460 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.459+0000 7f76e1ffb640 10 monclient: dump: 2024-03-25T18:21:35.460 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:21:35.460 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:21:35.461 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:21:35.461 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:21:35.461 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:35.461 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:21:35.461 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:21:35.461 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:21:35.461 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:21:35.461 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:21:35.462 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.460+0000 7f76e9582640 10 monclient: shutdown 2024-03-25T18:21:35.462 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.460+0000 7f76e9582640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:21:35.463 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.460+0000 7f76e9582640 1 -- 172.21.15.97:0/1914405770 >> v1:172.21.15.97:6790/0 conn(0x7f76e413f710 legacy=0x7f76e41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:35.463 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.460+0000 7f76e9582640 1 -- 172.21.15.97:0/1914405770 shutdown_connections 2024-03-25T18:21:35.463 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.460+0000 7f76e9582640 1 -- 172.21.15.97:0/1914405770 >> 172.21.15.97:0/1914405770 conn(0x7f76e40b7ea0 msgr2=0x7f76e40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:21:35.463 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.460+0000 7f76e9582640 1 -- 172.21.15.97:0/1914405770 shutdown_connections 2024-03-25T18:21:35.463 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.460+0000 7f76e9582640 1 -- 172.21.15.97:0/1914405770 wait complete. 2024-03-25T18:21:35.463 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.461+0000 7f76e9582640 10 monclient: build_initial_monmap 2024-03-25T18:21:35.463 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.461+0000 7f76e9582640 10 monclient: monmap: 2024-03-25T18:21:35.463 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:21:35.463 INFO:teuthology.misc.health.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:21:35.463 INFO:teuthology.misc.health.smithi097.stderr:last_changed 0.000000 2024-03-25T18:21:35.463 INFO:teuthology.misc.health.smithi097.stderr:created 0.000000 2024-03-25T18:21:35.463 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:21:35.463 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 1 2024-03-25T18:21:35.463 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:21:35.463 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T18:21:35.464 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-c 2024-03-25T18:21:35.464 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:21:35.464 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.462+0000 7f76e9582640 1 librados: starting msgr at 2024-03-25T18:21:35.464 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.462+0000 7f76e9582640 1 librados: starting objecter 2024-03-25T18:21:35.464 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.462+0000 7f76e9582640 1 Processor -- start 2024-03-25T18:21:35.464 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.462+0000 7f76e9582640 1 -- start start 2024-03-25T18:21:35.464 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.462+0000 7f76e9582640 1 librados: setting wanted keys 2024-03-25T18:21:35.464 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.462+0000 7f76e9582640 1 librados: calling monclient init 2024-03-25T18:21:35.464 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.462+0000 7f76e9582640 10 monclient: init 2024-03-25T18:21:35.465 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.463+0000 7f76e9582640 10 monclient: _reopen_session rank -1 2024-03-25T18:21:35.465 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.463+0000 7f76e9582640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:21:35.465 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.463+0000 7f76e9582640 10 monclient(hunting): picked mon.noname-a con 0x7f76e40a98f0 addr v1:172.21.15.97:6789/0 2024-03-25T18:21:35.465 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.463+0000 7f76e9582640 10 monclient(hunting): picked mon.noname-b con 0x7f76e413f710 addr v1:172.21.15.98:6789/0 2024-03-25T18:21:35.466 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.463+0000 7f76e9582640 10 monclient(hunting): picked mon.noname-c con 0x7f76e41430c0 addr v1:172.21.15.97:6790/0 2024-03-25T18:21:35.466 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.463+0000 7f76e9582640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76e4138700 con 0x7f76e40a98f0 2024-03-25T18:21:35.466 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.463+0000 7f76e9582640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76e4138b90 con 0x7f76e413f710 2024-03-25T18:21:35.466 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.463+0000 7f76e9582640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76e40b16d0 con 0x7f76e41430c0 2024-03-25T18:21:35.466 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.463+0000 7f76e9582640 10 monclient(hunting): _renew_subs 2024-03-25T18:21:35.466 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.463+0000 7f76e9582640 10 monclient(hunting): authenticate will time out at 7907.738310s 2024-03-25T18:21:35.474 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.474+0000 7f76e37fe640 1 --1- >> v1:172.21.15.97:6790/0 conn(0x7f76e41430c0 0x7f76e41e1dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.97:6790/0 says I am v1:172.21.15.97:43888/0 (socket says 172.21.15.97:43888) 2024-03-25T18:21:35.474 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.474+0000 7f76e37fe640 1 -- 172.21.15.97:0/2832103249 learned_addr learned my addr 172.21.15.97:0/2832103249 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:21:35.475 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.475+0000 7f76cb7fe640 10 client.?.objecter ms_handle_connect 0x7f76e41430c0 2024-03-25T18:21:35.475 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.475+0000 7f76cb7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:35.475 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.475+0000 7f76cb7fe640 10 client.?.objecter ms_handle_connect 0x7f76e413f710 2024-03-25T18:21:35.475 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.475+0000 7f76cb7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:35.483 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.483+0000 7f76cb7fe640 1 -- 172.21.15.97:0/2832103249 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1790449894 0 0) 0x7f76e4138b90 con 0x7f76e413f710 2024-03-25T18:21:35.483 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.483+0000 7f76cb7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:21:35.484 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.483+0000 7f76cb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:35.484 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.483+0000 7f76cb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:35.484 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.483+0000 7f76cb7fe640 10 monclient(hunting): my global_id is 4462 2024-03-25T18:21:35.484 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.483+0000 7f76cb7fe640 1 -- 172.21.15.97:0/2832103249 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76d0004eb0 con 0x7f76e413f710 2024-03-25T18:21:35.484 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.484+0000 7f76cb7fe640 1 -- 172.21.15.97:0/2832103249 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2919087195 0 0) 0x7f76e40b16d0 con 0x7f76e41430c0 2024-03-25T18:21:35.484 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.484+0000 7f76cb7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:21:35.484 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.484+0000 7f76cb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:35.484 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.484+0000 7f76cb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:35.485 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.484+0000 7f76cb7fe640 10 monclient(hunting): my global_id is 4454 2024-03-25T18:21:35.485 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.484+0000 7f76cb7fe640 1 -- 172.21.15.97:0/2832103249 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76e4138b90 con 0x7f76e41430c0 2024-03-25T18:21:35.488 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.488+0000 7f76cb7fe640 10 client.?.objecter ms_handle_connect 0x7f76e40a98f0 2024-03-25T18:21:35.488 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.488+0000 7f76cb7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:35.491 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.491+0000 7f76cb7fe640 1 -- 172.21.15.97:0/2832103249 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3908797273 0 0) 0x7f76d0004eb0 con 0x7f76e413f710 2024-03-25T18:21:35.491 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.491+0000 7f76cb7fe640 1 -- 172.21.15.97:0/2832103249 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f76e40b16d0 con 0x7f76e413f710 2024-03-25T18:21:35.491 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.491+0000 7f76cb7fe640 1 -- 172.21.15.97:0/2832103249 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f76d8001ba0 con 0x7f76e413f710 2024-03-25T18:21:35.491 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.491+0000 7f76cb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:35.492 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.491+0000 7f76cb7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:21:35.492 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.491+0000 7f76cb7fe640 10 monclient(hunting): dump: 2024-03-25T18:21:35.492 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:21:35.492 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:21:35.492 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:21:35.492 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:21:35.492 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:35.492 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:21:35.492 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:21:35.492 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:21:35.492 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:21:35.492 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:21:35.492 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.491+0000 7f76cb7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:21:35.503 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.502+0000 7f76cb7fe640 1 -- 172.21.15.97:0/2832103249 <== mon.1 v1:172.21.15.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3713984939 0 0) 0x7f76e40b16d0 con 0x7f76e413f710 2024-03-25T18:21:35.503 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.502+0000 7f76cb7fe640 1 -- 172.21.15.97:0/2832103249 >> v1:172.21.15.97:6790/0 conn(0x7f76e41430c0 legacy=0x7f76e41e1dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:35.503 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.502+0000 7f76cb7fe640 1 -- 172.21.15.97:0/2832103249 >> v1:172.21.15.97:6789/0 conn(0x7f76e40a98f0 legacy=0x7f76e4137820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:35.503 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.502+0000 7f76cb7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:21:35.503 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.502+0000 7f76cb7fe640 1 monclient: found mon.b 2024-03-25T18:21:35.503 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.502+0000 7f76cb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:21:35.503 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.502+0000 7f76cb7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:21:35.503 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.502+0000 7f76cb7fe640 1 -- 172.21.15.97:0/2832103249 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76e4138050 con 0x7f76e413f710 2024-03-25T18:21:35.503 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.502+0000 7f76cb7fe640 10 monclient: _finish_auth 0 2024-03-25T18:21:35.503 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.502+0000 7f76cb7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:21:35.504 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.502+0000 7f76cb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:21:35.504 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.502+0000 7f76e9582640 5 monclient: authenticate success, global_id 4462 2024-03-25T18:21:35.504 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.502+0000 7f76e9582640 10 monclient: _renew_subs 2024-03-25T18:21:35.504 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.502+0000 7f76e9582640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:21:35.504 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.502+0000 7f76e9582640 1 -- 172.21.15.97:0/2832103249 --> v1:172.21.15.98:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76e41382a0 con 0x7f76e413f710 2024-03-25T18:21:35.504 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.502+0000 7f76e9582640 10 client.4462.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:21:35.504 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.502+0000 7f76e9582640 10 monclient: _renew_subs 2024-03-25T18:21:35.504 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.502+0000 7f76e9582640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:21:35.504 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.502+0000 7f76e9582640 1 -- 172.21.15.97:0/2832103249 --> v1:172.21.15.98:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f76e41db170 con 0x7f76e413f710 2024-03-25T18:21:35.504 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.502+0000 7f76e9582640 1 librados: init done 2024-03-25T18:21:35.505 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.503+0000 7f76e9582640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:35.505 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.503+0000 7f76e9582640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:35.505 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.503+0000 7f76e9582640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:21:35.505 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.503+0000 7f76e9582640 1 -- 172.21.15.97:0/2832103249 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76e40b3ef0 con 0x7f76e413f710 2024-03-25T18:21:35.773 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.773+0000 7f76cb7fe640 1 -- 172.21.15.97:0/2832103249 <== mon.1 v1:172.21.15.98:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f76d8003910 con 0x7f76e413f710 2024-03-25T18:21:35.773 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.773+0000 7f76cb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:21:35.774 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.773+0000 7f76cb7fe640 1 -- 172.21.15.97:0/2832103249 <== mon.1 v1:172.21.15.98:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f76d8003ca0 con 0x7f76e413f710 2024-03-25T18:21:35.774 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.773+0000 7f76cb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:35.774 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.773+0000 7f76cb7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:21:35.774 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.773+0000 7f76cb7fe640 10 monclient: dump: 2024-03-25T18:21:35.774 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:21:35.774 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:21:35.774 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:21:35.774 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:21:35.774 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:35.774 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:21:35.774 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:21:35.774 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:21:35.774 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:21:35.774 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:21:35.774 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.773+0000 7f76e17fa640 4 set_mon_vals no callback set 2024-03-25T18:21:35.775 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.774+0000 7f76cb7fe640 1 -- 172.21.15.97:0/2832103249 <== mon.1 v1:172.21.15.98:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2377622137 0 0) 0x7f76d8003ec0 con 0x7f76e413f710 2024-03-25T18:21:35.775 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.775+0000 7f76e2ffd640 10 monclient: get_auth_request con 0x7f76d006dd10 auth_method 0 2024-03-25T18:21:35.775 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.775+0000 7f76cb7fe640 10 client.4462.objecter ms_handle_connect 0x7f76d006dd10 2024-03-25T18:21:35.777 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.777+0000 7f76cb7fe640 1 -- 172.21.15.97:0/2832103249 <== mon.1 v1:172.21.15.98:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 430711918 0 0) 0x7f76d80536b0 con 0x7f76e413f710 2024-03-25T18:21:35.778 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.777+0000 7f76cb7fe640 10 client.4462.objecter ms_dispatch 0x7f76e4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:21:35.778 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.777+0000 7f76cb7fe640 3 client.4462.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:21:35.778 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.777+0000 7f76cb7fe640 3 client.4462.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:21:35.778 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.777+0000 7f76cb7fe640 20 client.4462.objecter dump_active .. 0 homeless 2024-03-25T18:21:35.781 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.780+0000 7f76cb7fe640 1 -- 172.21.15.97:0/2832103249 <== mon.1 v1:172.21.15.98:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f76d804f1c0 con 0x7f76e413f710 2024-03-25T18:21:35.781 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.781+0000 7f76cb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:35.781 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:35.781+0000 7f76cb7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:21:36.026 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:36.025+0000 7f76e9582640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:36.026 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:36.025+0000 7f76e9582640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:36.026 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:36.025+0000 7f76e9582640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:21:36.026 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:36.025+0000 7f76e9582640 1 -- 172.21.15.97:0/2832103249 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f76e41384c0 con 0x7f76e413f710 2024-03-25T18:21:36.051 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:36.050+0000 7f76cb7fe640 1 -- 172.21.15.97:0/2832103249 <== mon.1 v1:172.21.15.98:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2169427412) 0x7f76d80510f0 con 0x7f76e413f710 2024-03-25T18:21:36.051 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:36.050+0000 7f76cb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:36.051 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:36.050+0000 7f76cb7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:21:36.051 INFO:teuthology.misc.health.smithi097.stdout: 2024-03-25T18:21:36.052 INFO:teuthology.misc.health.smithi097.stdout:{"epoch":10,"fsid":"4c3d472c-79b8-486b-9e3d-2738d4306754","created":"2024-03-25T18:16:14.737309+0000","modified":"2024-03-25T18:16:20.357492+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:20.357492+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cb5152b9-262f-45c3-94b8-18d0e81df558","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6daf8793-0838-473d-a7d6-9a0d2894c75d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"136f1493-3b9c-4f43-a2e7-641482d69dae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"397d538c-4f64-4daf-9e8b-662d09f91327","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4f1df08-663f-4bda-93f8-3878ab7c24b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c4b18d70-a3ce-4b15-9b70-eae5525bd761","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8dd4033a-de0e-4f8a-96dc-59f3c1f825b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0077722e-ffa6-4a3d-9891-929d9869b286","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:21:36.054 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:36.053+0000 7f76e9582640 10 librados: watch_flush enter 2024-03-25T18:21:36.054 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:36.053+0000 7f76e9582640 10 librados: watch_flush exit 2024-03-25T18:21:36.054 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:36.053+0000 7f76e9582640 20 client.4462.objecter shutdown clearing up homeless session... 2024-03-25T18:21:36.054 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:36.053+0000 7f76e9582640 10 client.4462.objecter successfully canceled tick 2024-03-25T18:21:36.054 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:36.053+0000 7f76e9582640 1 -- 172.21.15.97:0/2832103249 >> v1:172.21.15.98:6800/800387573 conn(0x7f76d006dd10 legacy=0x7f76d00701d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:36.054 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:36.053+0000 7f76e9582640 10 monclient: shutdown 2024-03-25T18:21:36.054 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:36.053+0000 7f76e9582640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:21:36.054 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:36.053+0000 7f76e9582640 1 -- 172.21.15.97:0/2832103249 >> v1:172.21.15.98:6789/0 conn(0x7f76e413f710 legacy=0x7f76e41da880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:36.054 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:36.053+0000 7f76e9582640 1 -- 172.21.15.97:0/2832103249 shutdown_connections 2024-03-25T18:21:36.055 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:36.053+0000 7f76e9582640 1 -- 172.21.15.97:0/2832103249 >> 172.21.15.97:0/2832103249 conn(0x7f76e40b7ea0 msgr2=0x7f76e4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:21:36.055 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:36.053+0000 7f76e9582640 1 -- 172.21.15.97:0/2832103249 shutdown_connections 2024-03-25T18:21:36.055 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:36.053+0000 7f76e9582640 1 -- 172.21.15.97:0/2832103249 wait complete. 2024-03-25T18:21:36.055 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:36.053+0000 7f76e9582640 1 librados: shutdown 2024-03-25T18:21:36.067 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:21:42.874 DEBUG:teuthology.orchestra.run.smithi097:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:21:42.988 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:42.986+0000 7f10ddeb5640 10 monclient: get_monmap_and_config 2024-03-25T18:21:42.988 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:42.986+0000 7f10ddeb5640 10 monclient: build_initial_monmap 2024-03-25T18:21:42.988 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:42.986+0000 7f10ddeb5640 10 monclient: monmap: 2024-03-25T18:21:42.988 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:21:42.988 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:21:42.988 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:21:42.988217+0000 2024-03-25T18:21:42.988 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:21:42.988217+0000 2024-03-25T18:21:42.989 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:21:42.989 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:21:42.989 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:21:42.989 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T18:21:42.989 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-b 2024-03-25T18:21:42.989 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:21:42.990 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:42.987+0000 7f10ddeb5640 1 Processor -- start 2024-03-25T18:21:42.990 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:42.987+0000 7f10ddeb5640 1 -- start start 2024-03-25T18:21:42.990 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:42.987+0000 7f10ddeb5640 10 monclient: init 2024-03-25T18:21:42.990 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:42.988+0000 7f10ddeb5640 10 monclient: _reopen_session rank -1 2024-03-25T18:21:42.990 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:42.988+0000 7f10ddeb5640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:21:42.991 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:42.988+0000 7f10ddeb5640 10 monclient(hunting): picked mon.noname-c con 0x7f10d80b4d60 addr v1:172.21.15.98:6789/0 2024-03-25T18:21:42.991 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:42.988+0000 7f10ddeb5640 10 monclient(hunting): picked mon.noname-a con 0x7f10d80b1730 addr v1:172.21.15.97:6789/0 2024-03-25T18:21:42.991 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:42.988+0000 7f10ddeb5640 10 monclient(hunting): picked mon.noname-b con 0x7f10d814be90 addr v1:172.21.15.97:6790/0 2024-03-25T18:21:42.991 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:42.988+0000 7f10ddeb5640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10d80af940 con 0x7f10d80b1730 2024-03-25T18:21:42.991 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:42.988+0000 7f10ddeb5640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10d80b0a70 con 0x7f10d80b4d60 2024-03-25T18:21:42.991 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:42.988+0000 7f10ddeb5640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10d80b0d50 con 0x7f10d814be90 2024-03-25T18:21:42.991 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:42.988+0000 7f10ddeb5640 10 monclient(hunting): _renew_subs 2024-03-25T18:21:42.991 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:42.988+0000 7f10ddeb5640 10 monclient(hunting): authenticate will time out at 7915.264444s 2024-03-25T18:21:43.007 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.006+0000 7f10d6ffd640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f10d80b1730 0x7f10d80b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:59988/0 (socket says 172.21.15.97:59988) 2024-03-25T18:21:43.007 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.006+0000 7f10d6ffd640 1 -- 172.21.15.97:0/724232266 learned_addr learned my addr 172.21.15.97:0/724232266 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:21:43.021 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.020+0000 7f10d67fc640 1 -- 172.21.15.97:0/724232266 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 305060269 0 0) 0x7f10d80b0a70 con 0x7f10d80b4d60 2024-03-25T18:21:43.021 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.020+0000 7f10d67fc640 10 monclient(hunting): _negotiate 2024-03-25T18:21:43.021 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.020+0000 7f10d67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:43.021 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.020+0000 7f10d67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:43.021 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.020+0000 7f10d67fc640 10 monclient(hunting): my global_id is 4465 2024-03-25T18:21:43.022 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.020+0000 7f10d67fc640 1 -- 172.21.15.97:0/724232266 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10c0003520 con 0x7f10d80b4d60 2024-03-25T18:21:43.029 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.027+0000 7f10d67fc640 1 -- 172.21.15.97:0/724232266 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2723427298 0 0) 0x7f10c0003520 con 0x7f10d80b4d60 2024-03-25T18:21:43.029 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.027+0000 7f10d67fc640 1 -- 172.21.15.97:0/724232266 >> v1:172.21.15.97:6790/0 conn(0x7f10d814be90 legacy=0x7f10d814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:43.029 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.027+0000 7f10d67fc640 1 -- 172.21.15.97:0/724232266 >> v1:172.21.15.97:6789/0 conn(0x7f10d80b1730 legacy=0x7f10d80b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:43.029 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.027+0000 7f10d67fc640 10 monclient: _finish_hunting 0 2024-03-25T18:21:43.029 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.027+0000 7f10d67fc640 1 monclient: found mon.noname-c 2024-03-25T18:21:43.029 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.027+0000 7f10d67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:21:43.029 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.027+0000 7f10d67fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.98:6789/0 2024-03-25T18:21:43.029 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.027+0000 7f10d67fc640 1 -- 172.21.15.97:0/724232266 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10d80b1030 con 0x7f10d80b4d60 2024-03-25T18:21:43.029 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.027+0000 7f10d67fc640 10 monclient: _finish_auth 0 2024-03-25T18:21:43.030 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.027+0000 7f10d67fc640 10 monclient: _check_auth_tickets 2024-03-25T18:21:43.030 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.027+0000 7f10d67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:21:43.030 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.027+0000 7f10d67fc640 1 -- 172.21.15.97:0/724232266 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f10c8001690 con 0x7f10d80b4d60 2024-03-25T18:21:43.030 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.027+0000 7f10ddeb5640 5 monclient: authenticate success, global_id 4465 2024-03-25T18:21:43.030 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.027+0000 7f10ddeb5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:21:43.030 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.027+0000 7f10d67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:43.030 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.027+0000 7f10d67fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:21:43.030 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.027+0000 7f10d67fc640 10 monclient: dump: 2024-03-25T18:21:43.030 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:21:43.030 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:21:43.030 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:21:43.030 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:21:43.030 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:43.031 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:21:43.031 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:21:43.031 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:21:43.031 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:21:43.031 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:21:43.031 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.027+0000 7f10ddeb5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:21:43.036 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.034+0000 7f10d67fc640 1 -- 172.21.15.97:0/724232266 <== mon.1 v1:172.21.15.98:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f10c80017f0 con 0x7f10d80b4d60 2024-03-25T18:21:43.036 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.034+0000 7f10d67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:21:43.036 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.034+0000 7f10d67fc640 1 -- 172.21.15.97:0/724232266 <== mon.1 v1:172.21.15.98:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f10c8001c00 con 0x7f10d80b4d60 2024-03-25T18:21:43.036 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.034+0000 7f10ddeb5640 10 monclient: get_monmap_and_config success 2024-03-25T18:21:43.036 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.034+0000 7f10ddeb5640 4 set_mon_vals no callback set 2024-03-25T18:21:43.036 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.034+0000 7f10d67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:43.036 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.034+0000 7f10d67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:21:43.036 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.034+0000 7f10d67fc640 10 monclient: dump: 2024-03-25T18:21:43.037 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:21:43.037 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:21:43.037 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:21:43.037 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:21:43.037 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:43.037 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:21:43.037 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:21:43.037 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:21:43.037 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:21:43.037 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:21:43.039 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.035+0000 7f10ddeb5640 10 monclient: shutdown 2024-03-25T18:21:43.039 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.035+0000 7f10ddeb5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:21:43.039 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.035+0000 7f10ddeb5640 1 -- 172.21.15.97:0/724232266 >> v1:172.21.15.98:6789/0 conn(0x7f10d80b4d60 legacy=0x7f10d813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:43.039 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.035+0000 7f10ddeb5640 1 -- 172.21.15.97:0/724232266 shutdown_connections 2024-03-25T18:21:43.039 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.035+0000 7f10ddeb5640 1 -- 172.21.15.97:0/724232266 >> 172.21.15.97:0/724232266 conn(0x7f10d80617a0 msgr2=0x7f10d8061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:21:43.039 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.036+0000 7f10ddeb5640 1 -- 172.21.15.97:0/724232266 shutdown_connections 2024-03-25T18:21:43.039 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.036+0000 7f10ddeb5640 1 -- 172.21.15.97:0/724232266 wait complete. 2024-03-25T18:21:43.039 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.036+0000 7f10ddeb5640 10 monclient: build_initial_monmap 2024-03-25T18:21:43.039 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.036+0000 7f10ddeb5640 10 monclient: monmap: 2024-03-25T18:21:43.040 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:21:43.040 INFO:teuthology.misc.health.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:21:43.040 INFO:teuthology.misc.health.smithi097.stderr:last_changed 0.000000 2024-03-25T18:21:43.040 INFO:teuthology.misc.health.smithi097.stderr:created 0.000000 2024-03-25T18:21:43.040 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:21:43.040 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 1 2024-03-25T18:21:43.040 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:21:43.040 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T18:21:43.040 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-c 2024-03-25T18:21:43.040 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:21:43.040 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.037+0000 7f10ddeb5640 1 librados: starting msgr at 2024-03-25T18:21:43.040 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.037+0000 7f10ddeb5640 1 librados: starting objecter 2024-03-25T18:21:43.040 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.037+0000 7f10ddeb5640 1 Processor -- start 2024-03-25T18:21:43.040 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.037+0000 7f10ddeb5640 1 -- start start 2024-03-25T18:21:43.040 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.037+0000 7f10ddeb5640 1 librados: setting wanted keys 2024-03-25T18:21:43.041 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.037+0000 7f10ddeb5640 1 librados: calling monclient init 2024-03-25T18:21:43.041 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.037+0000 7f10ddeb5640 10 monclient: init 2024-03-25T18:21:43.042 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.038+0000 7f10ddeb5640 10 monclient: _reopen_session rank -1 2024-03-25T18:21:43.042 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.038+0000 7f10ddeb5640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:21:43.042 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.038+0000 7f10ddeb5640 10 monclient(hunting): picked mon.noname-b con 0x7f10d80b1730 addr v1:172.21.15.98:6789/0 2024-03-25T18:21:43.042 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.038+0000 7f10ddeb5640 10 monclient(hunting): picked mon.noname-c con 0x7f10d80b4d60 addr v1:172.21.15.97:6790/0 2024-03-25T18:21:43.042 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.038+0000 7f10ddeb5640 10 monclient(hunting): picked mon.noname-a con 0x7f10d814be90 addr v1:172.21.15.97:6789/0 2024-03-25T18:21:43.042 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.038+0000 7f10ddeb5640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10d8138410 con 0x7f10d814be90 2024-03-25T18:21:43.042 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.038+0000 7f10ddeb5640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10d81dad50 con 0x7f10d80b1730 2024-03-25T18:21:43.042 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.038+0000 7f10ddeb5640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10d81db980 con 0x7f10d80b4d60 2024-03-25T18:21:43.042 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.038+0000 7f10ddeb5640 10 monclient(hunting): _renew_subs 2024-03-25T18:21:43.042 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.038+0000 7f10ddeb5640 10 monclient(hunting): authenticate will time out at 7915.314644s 2024-03-25T18:21:43.047 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.045+0000 7f10d77fe640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f10d80b1730 0x7f10d81db270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.97:38180/0 (socket says 172.21.15.97:38180) 2024-03-25T18:21:43.047 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.045+0000 7f10d77fe640 1 -- 172.21.15.97:0/2970636602 learned_addr learned my addr 172.21.15.97:0/2970636602 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:21:43.048 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.046+0000 7f10bffff640 10 client.?.objecter ms_handle_connect 0x7f10d80b1730 2024-03-25T18:21:43.048 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.046+0000 7f10bffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:43.051 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.049+0000 7f10bffff640 10 client.?.objecter ms_handle_connect 0x7f10d80b4d60 2024-03-25T18:21:43.051 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.049+0000 7f10bffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:43.058 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.056+0000 7f10bffff640 10 client.?.objecter ms_handle_connect 0x7f10d814be90 2024-03-25T18:21:43.058 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.056+0000 7f10bffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:43.058 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.056+0000 7f10bffff640 1 -- 172.21.15.97:0/2970636602 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1206469278 0 0) 0x7f10d81db980 con 0x7f10d80b4d60 2024-03-25T18:21:43.058 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.056+0000 7f10bffff640 10 monclient(hunting): _negotiate 2024-03-25T18:21:43.058 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.056+0000 7f10bffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:43.058 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.056+0000 7f10bffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:43.058 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.056+0000 7f10bffff640 10 monclient(hunting): my global_id is 4460 2024-03-25T18:21:43.058 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.056+0000 7f10bffff640 1 -- 172.21.15.97:0/2970636602 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10c4004eb0 con 0x7f10d80b4d60 2024-03-25T18:21:43.070 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.068+0000 7f10bffff640 1 -- 172.21.15.97:0/2970636602 <== mon.2 v1:172.21.15.97:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3634754280 0 0) 0x7f10c4004eb0 con 0x7f10d80b4d60 2024-03-25T18:21:43.070 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.068+0000 7f10bffff640 1 -- 172.21.15.97:0/2970636602 --> v1:172.21.15.97:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f10d81db980 con 0x7f10d80b4d60 2024-03-25T18:21:43.070 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.068+0000 7f10bffff640 1 -- 172.21.15.97:0/2970636602 <== mon.2 v1:172.21.15.97:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f10cc0018a0 con 0x7f10d80b4d60 2024-03-25T18:21:43.070 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.068+0000 7f10bffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:43.070 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.068+0000 7f10bffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:21:43.070 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.068+0000 7f10bffff640 10 monclient(hunting): dump: 2024-03-25T18:21:43.070 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:21:43.070 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:21:43.070 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:21:43.070 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:21:43.070 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:43.071 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:21:43.071 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:21:43.071 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:21:43.071 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:21:43.071 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:21:43.071 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.068+0000 7f10bffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:21:43.073 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.071+0000 7f10bffff640 1 -- 172.21.15.97:0/2970636602 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3091664530 0 0) 0x7f10d81dad50 con 0x7f10d80b1730 2024-03-25T18:21:43.073 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.071+0000 7f10bffff640 10 monclient(hunting): _negotiate 2024-03-25T18:21:43.073 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.071+0000 7f10bffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:43.073 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.071+0000 7f10bffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:43.073 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.071+0000 7f10bffff640 10 monclient(hunting): my global_id is 4468 2024-03-25T18:21:43.073 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.071+0000 7f10bffff640 1 -- 172.21.15.97:0/2970636602 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10c4004eb0 con 0x7f10d80b1730 2024-03-25T18:21:43.081 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.079+0000 7f10bffff640 1 -- 172.21.15.97:0/2970636602 <== mon.2 v1:172.21.15.97:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2779218529 0 0) 0x7f10d81db980 con 0x7f10d80b4d60 2024-03-25T18:21:43.081 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.079+0000 7f10bffff640 1 -- 172.21.15.97:0/2970636602 >> v1:172.21.15.98:6789/0 conn(0x7f10d80b1730 legacy=0x7f10d81db270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:43.081 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.079+0000 7f10bffff640 1 -- 172.21.15.97:0/2970636602 >> v1:172.21.15.97:6789/0 conn(0x7f10d814be90 legacy=0x7f10d81e2ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:43.081 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.079+0000 7f10bffff640 10 monclient: _finish_hunting 0 2024-03-25T18:21:43.081 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.079+0000 7f10bffff640 1 monclient: found mon.c 2024-03-25T18:21:43.081 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.079+0000 7f10bffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:21:43.082 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.079+0000 7f10bffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.97:6790/0 2024-03-25T18:21:43.082 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.079+0000 7f10bffff640 1 -- 172.21.15.97:0/2970636602 --> v1:172.21.15.97:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10d8136090 con 0x7f10d80b4d60 2024-03-25T18:21:43.082 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.079+0000 7f10bffff640 10 monclient: _finish_auth 0 2024-03-25T18:21:43.082 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.079+0000 7f10bffff640 10 monclient: _check_auth_tickets 2024-03-25T18:21:43.082 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.079+0000 7f10bffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:21:43.082 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.079+0000 7f10ddeb5640 5 monclient: authenticate success, global_id 4460 2024-03-25T18:21:43.082 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.079+0000 7f10ddeb5640 10 monclient: _renew_subs 2024-03-25T18:21:43.082 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.079+0000 7f10ddeb5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.97:6790/0 2024-03-25T18:21:43.082 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.079+0000 7f10ddeb5640 1 -- 172.21.15.97:0/2970636602 --> v1:172.21.15.97:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10d8136470 con 0x7f10d80b4d60 2024-03-25T18:21:43.082 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.079+0000 7f10ddeb5640 10 client.4460.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:21:43.082 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.079+0000 7f10ddeb5640 10 monclient: _renew_subs 2024-03-25T18:21:43.082 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.079+0000 7f10ddeb5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.97:6790/0 2024-03-25T18:21:43.082 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.079+0000 7f10ddeb5640 1 -- 172.21.15.97:0/2970636602 --> v1:172.21.15.97:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f10d8143ae0 con 0x7f10d80b4d60 2024-03-25T18:21:43.082 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.079+0000 7f10ddeb5640 1 librados: init done 2024-03-25T18:21:43.083 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.080+0000 7f10ddeb5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:43.083 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.080+0000 7f10ddeb5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:43.083 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.080+0000 7f10ddeb5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.97:6790/0 2024-03-25T18:21:43.083 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.080+0000 7f10ddeb5640 1 -- 172.21.15.97:0/2970636602 --> v1:172.21.15.97:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10d81daee0 con 0x7f10d80b4d60 2024-03-25T18:21:43.372 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.370+0000 7f10bffff640 1 -- 172.21.15.97:0/2970636602 <== mon.2 v1:172.21.15.97:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f10cc001d50 con 0x7f10d80b4d60 2024-03-25T18:21:43.372 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.370+0000 7f10bffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:21:43.372 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.370+0000 7f10bffff640 1 -- 172.21.15.97:0/2970636602 <== mon.2 v1:172.21.15.97:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f10cc004a00 con 0x7f10d80b4d60 2024-03-25T18:21:43.372 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.370+0000 7f10bffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:43.372 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.370+0000 7f10bffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:21:43.372 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.370+0000 7f10bffff640 10 monclient: dump: 2024-03-25T18:21:43.372 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:21:43.372 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:21:43.372 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:21:43.372 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:21:43.372 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:43.373 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:21:43.373 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:21:43.373 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:21:43.373 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:21:43.373 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:21:43.373 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.370+0000 7f10d5ffb640 4 set_mon_vals no callback set 2024-03-25T18:21:43.373 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.371+0000 7f10bffff640 1 -- 172.21.15.97:0/2970636602 <== mon.2 v1:172.21.15.97:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2377622137 0 0) 0x7f10cc004c20 con 0x7f10d80b4d60 2024-03-25T18:21:43.374 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.372+0000 7f10d77fe640 10 monclient: get_auth_request con 0x7f10c406dda0 auth_method 0 2024-03-25T18:21:43.374 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.372+0000 7f10bffff640 10 client.4460.objecter ms_handle_connect 0x7f10c406dda0 2024-03-25T18:21:43.378 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.376+0000 7f10bffff640 1 -- 172.21.15.97:0/2970636602 <== mon.2 v1:172.21.15.97:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 430711918 0 0) 0x7f10cc0547a0 con 0x7f10d80b4d60 2024-03-25T18:21:43.378 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.376+0000 7f10bffff640 10 client.4460.objecter ms_dispatch 0x7f10d8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:21:43.378 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.376+0000 7f10bffff640 3 client.4460.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:21:43.378 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.376+0000 7f10bffff640 3 client.4460.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:21:43.378 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.376+0000 7f10bffff640 20 client.4460.objecter dump_active .. 0 homeless 2024-03-25T18:21:43.378 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.376+0000 7f10bffff640 1 -- 172.21.15.97:0/2970636602 <== mon.2 v1:172.21.15.97:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f10cc0500e0 con 0x7f10d80b4d60 2024-03-25T18:21:43.378 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.376+0000 7f10bffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:43.378 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.376+0000 7f10bffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:21:43.614 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.613+0000 7f10ddeb5640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:43.615 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.613+0000 7f10ddeb5640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:43.615 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.613+0000 7f10ddeb5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.97:6790/0 2024-03-25T18:21:43.615 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.613+0000 7f10ddeb5640 1 -- 172.21.15.97:0/2970636602 --> v1:172.21.15.97:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f10d8136220 con 0x7f10d80b4d60 2024-03-25T18:21:43.693 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.691+0000 7f10bffff640 1 -- 172.21.15.97:0/2970636602 <== mon.2 v1:172.21.15.97:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2169427412) 0x7f10cc0521e0 con 0x7f10d80b4d60 2024-03-25T18:21:43.693 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.691+0000 7f10bffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:43.693 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.691+0000 7f10bffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:21:43.693 INFO:teuthology.misc.health.smithi097.stdout: 2024-03-25T18:21:43.693 INFO:teuthology.misc.health.smithi097.stdout:{"epoch":10,"fsid":"4c3d472c-79b8-486b-9e3d-2738d4306754","created":"2024-03-25T18:16:14.737309+0000","modified":"2024-03-25T18:16:20.357492+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:20.357492+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cb5152b9-262f-45c3-94b8-18d0e81df558","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6daf8793-0838-473d-a7d6-9a0d2894c75d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"136f1493-3b9c-4f43-a2e7-641482d69dae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"397d538c-4f64-4daf-9e8b-662d09f91327","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4f1df08-663f-4bda-93f8-3878ab7c24b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c4b18d70-a3ce-4b15-9b70-eae5525bd761","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8dd4033a-de0e-4f8a-96dc-59f3c1f825b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0077722e-ffa6-4a3d-9891-929d9869b286","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:21:43.696 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.694+0000 7f10ddeb5640 10 librados: watch_flush enter 2024-03-25T18:21:43.696 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.694+0000 7f10ddeb5640 10 librados: watch_flush exit 2024-03-25T18:21:43.696 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.694+0000 7f10ddeb5640 20 client.4460.objecter shutdown clearing up homeless session... 2024-03-25T18:21:43.696 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.694+0000 7f10ddeb5640 10 client.4460.objecter successfully canceled tick 2024-03-25T18:21:43.696 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.694+0000 7f10ddeb5640 1 -- 172.21.15.97:0/2970636602 >> v1:172.21.15.98:6800/800387573 conn(0x7f10c406dda0 legacy=0x7f10c4070260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:43.696 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.694+0000 7f10ddeb5640 10 monclient: shutdown 2024-03-25T18:21:43.696 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.694+0000 7f10ddeb5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:21:43.696 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.694+0000 7f10ddeb5640 1 -- 172.21.15.97:0/2970636602 >> v1:172.21.15.97:6790/0 conn(0x7f10d80b4d60 legacy=0x7f10d812fcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:43.696 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.694+0000 7f10ddeb5640 1 -- 172.21.15.97:0/2970636602 shutdown_connections 2024-03-25T18:21:43.696 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.694+0000 7f10ddeb5640 1 -- 172.21.15.97:0/2970636602 >> 172.21.15.97:0/2970636602 conn(0x7f10d80617a0 msgr2=0x7f10d80b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:21:43.696 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.694+0000 7f10ddeb5640 1 -- 172.21.15.97:0/2970636602 shutdown_connections 2024-03-25T18:21:43.697 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.694+0000 7f10ddeb5640 1 -- 172.21.15.97:0/2970636602 wait complete. 2024-03-25T18:21:43.697 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:43.694+0000 7f10ddeb5640 1 librados: shutdown 2024-03-25T18:21:43.707 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:21:50.514 DEBUG:teuthology.orchestra.run.smithi097:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:21:50.631 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:50.629+0000 7f665feed640 10 monclient: get_monmap_and_config 2024-03-25T18:21:50.631 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:50.630+0000 7f665feed640 10 monclient: build_initial_monmap 2024-03-25T18:21:50.631 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:50.630+0000 7f665feed640 10 monclient: monmap: 2024-03-25T18:21:50.631 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:21:50.632 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:21:50.632 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:21:50.631289+0000 2024-03-25T18:21:50.632 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:21:50.631289+0000 2024-03-25T18:21:50.632 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:21:50.632 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:21:50.632 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:21:50.632 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T18:21:50.632 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-b 2024-03-25T18:21:50.632 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:21:50.633 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:50.630+0000 7f665feed640 1 Processor -- start 2024-03-25T18:21:50.633 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:50.631+0000 7f665feed640 1 -- start start 2024-03-25T18:21:50.633 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:50.631+0000 7f665feed640 10 monclient: init 2024-03-25T18:21:50.634 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:50.631+0000 7f665feed640 10 monclient: _reopen_session rank -1 2024-03-25T18:21:50.634 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:50.631+0000 7f665feed640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:21:50.634 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:50.631+0000 7f665feed640 10 monclient(hunting): picked mon.noname-a con 0x7f66580a98f0 addr v1:172.21.15.97:6789/0 2024-03-25T18:21:50.634 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:50.631+0000 7f665feed640 10 monclient(hunting): picked mon.noname-b con 0x7f665813f710 addr v1:172.21.15.97:6790/0 2024-03-25T18:21:50.634 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:50.631+0000 7f665feed640 10 monclient(hunting): picked mon.noname-c con 0x7f66581430c0 addr v1:172.21.15.98:6789/0 2024-03-25T18:21:50.634 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:50.631+0000 7f665feed640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6658142520 con 0x7f66580a98f0 2024-03-25T18:21:50.634 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:50.631+0000 7f665feed640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66580b28a0 con 0x7f66581430c0 2024-03-25T18:21:50.634 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:50.631+0000 7f665feed640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66580b2a30 con 0x7f665813f710 2024-03-25T18:21:50.634 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:50.631+0000 7f665feed640 10 monclient(hunting): _renew_subs 2024-03-25T18:21:50.634 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:50.631+0000 7f665feed640 10 monclient(hunting): authenticate will time out at 7922.907744s 2024-03-25T18:21:50.644 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:50.642+0000 7f665d488640 1 --1- >> v1:172.21.15.97:6790/0 conn(0x7f665813f710 0x7f66581429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6790/0 says I am v1:172.21.15.97:48696/0 (socket says 172.21.15.97:48696) 2024-03-25T18:21:50.644 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:50.642+0000 7f665d488640 1 -- 172.21.15.97:0/2642448369 learned_addr learned my addr 172.21.15.97:0/2642448369 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:21:50.656 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:50.655+0000 7f665cc87640 1 -- 172.21.15.97:0/2642448369 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1346144382 0 0) 0x7f66580b28a0 con 0x7f66581430c0 2024-03-25T18:21:50.656 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:50.655+0000 7f665cc87640 10 monclient(hunting): _negotiate 2024-03-25T18:21:50.656 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:50.655+0000 7f665cc87640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:50.656 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:50.655+0000 7f665cc87640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:50.656 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:50.655+0000 7f665cc87640 10 monclient(hunting): my global_id is 4471 2024-03-25T18:21:50.656 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:50.655+0000 7f665cc87640 1 -- 172.21.15.97:0/2642448369 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6644003520 con 0x7f66581430c0 2024-03-25T18:21:50.657 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:50.656+0000 7f665cc87640 1 -- 172.21.15.97:0/2642448369 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 524850733 0 0) 0x7f66580b2a30 con 0x7f665813f710 2024-03-25T18:21:50.657 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:50.656+0000 7f665cc87640 10 monclient(hunting): _negotiate 2024-03-25T18:21:50.658 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:50.656+0000 7f665cc87640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:50.658 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:50.656+0000 7f665cc87640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:50.658 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:50.656+0000 7f665cc87640 10 monclient(hunting): my global_id is 4463 2024-03-25T18:21:50.658 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:50.656+0000 7f665cc87640 1 -- 172.21.15.97:0/2642448369 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f66580b28a0 con 0x7f665813f710 2024-03-25T18:21:50.666 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:50.665+0000 7f665cc87640 1 -- 172.21.15.97:0/2642448369 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 149859124 0 0) 0x7f6644003520 con 0x7f66581430c0 2024-03-25T18:21:50.667 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:50.665+0000 7f665cc87640 1 -- 172.21.15.97:0/2642448369 >> v1:172.21.15.97:6790/0 conn(0x7f665813f710 legacy=0x7f66581429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:50.667 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:50.665+0000 7f665cc87640 1 -- 172.21.15.97:0/2642448369 >> v1:172.21.15.97:6789/0 conn(0x7f66580a98f0 legacy=0x7f66580a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:21:50.667 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:50.665+0000 7f665cc87640 10 monclient: _finish_hunting 0 2024-03-25T18:21:50.667 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:50.665+0000 7f665cc87640 1 monclient: found mon.noname-c 2024-03-25T18:21:50.667 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:50.665+0000 7f665cc87640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:21:50.667 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:50.665+0000 7f665cc87640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.98:6789/0 2024-03-25T18:21:50.667 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:50.665+0000 7f665cc87640 1 -- 172.21.15.97:0/2642448369 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66580b44c0 con 0x7f66581430c0 2024-03-25T18:21:50.667 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:50.665+0000 7f665cc87640 10 monclient: _finish_auth 0 2024-03-25T18:21:50.667 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:50.665+0000 7f665cc87640 10 monclient: _check_auth_tickets 2024-03-25T18:21:50.667 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:50.665+0000 7f665cc87640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:21:50.667 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:50.665+0000 7f665cc87640 1 -- 172.21.15.97:0/2642448369 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f6650001a10 con 0x7f66581430c0 2024-03-25T18:21:50.667 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:50.665+0000 7f665cc87640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:50.667 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:50.665+0000 7f665cc87640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:21:50.667 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:50.665+0000 7f665cc87640 10 monclient: dump: 2024-03-25T18:21:50.668 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:21:50.668 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:21:50.668 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:21:50.668 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:21:50.668 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:50.668 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:21:50.668 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:21:50.668 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:21:50.668 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:21:50.668 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:21:50.668 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:50.665+0000 7f665feed640 5 monclient: authenticate success, global_id 4471 2024-03-25T18:21:50.668 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:50.665+0000 7f665feed640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:21:50.674 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:50.673+0000 7f665cc87640 1 -- 172.21.15.97:0/2642448369 <== mon.1 v1:172.21.15.98:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6650001b70 con 0x7f66581430c0 2024-03-25T18:21:50.674 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:50.673+0000 7f665cc87640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:21:50.674 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:50.673+0000 7f665cc87640 1 -- 172.21.15.97:0/2642448369 <== mon.1 v1:172.21.15.98:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f6650001f80 con 0x7f66581430c0 2024-03-25T18:21:50.674 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:50.673+0000 7f665feed640 10 monclient: get_monmap_and_config success 2024-03-25T18:21:50.674 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:50.673+0000 7f665feed640 4 set_mon_vals no callback set 2024-03-25T18:21:50.674 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:50.673+0000 7f665cc87640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:50.675 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:50.673+0000 7f665cc87640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:21:50.675 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:50.673+0000 7f665cc87640 10 monclient: dump: 2024-03-25T18:21:50.675 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:21:50.675 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:21:50.675 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:21:50.675 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:21:50.675 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:50.675 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:21:50.675 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:21:50.675 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:21:50.675 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:21:50.675 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:21:50.677 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:50.673+0000 7f665feed640 10 monclient: shutdown 2024-03-25T18:21:50.677 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:50.673+0000 7f665feed640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:21:50.677 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:50.673+0000 7f665feed640 1 -- 172.21.15.97:0/2642448369 >> v1:172.21.15.98:6789/0 conn(0x7f66581430c0 legacy=0x7f665814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:50.677 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:50.674+0000 7f665feed640 1 -- 172.21.15.97:0/2642448369 shutdown_connections 2024-03-25T18:21:50.677 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:50.674+0000 7f665feed640 1 -- 172.21.15.97:0/2642448369 >> 172.21.15.97:0/2642448369 conn(0x7f66580b7ea0 msgr2=0x7f66580b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:21:50.677 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:50.674+0000 7f665feed640 1 -- 172.21.15.97:0/2642448369 shutdown_connections 2024-03-25T18:21:50.677 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:50.674+0000 7f665feed640 1 -- 172.21.15.97:0/2642448369 wait complete. 2024-03-25T18:21:50.677 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:50.674+0000 7f665feed640 10 monclient: build_initial_monmap 2024-03-25T18:21:50.677 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:50.674+0000 7f665feed640 10 monclient: monmap: 2024-03-25T18:21:50.677 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:21:50.677 INFO:teuthology.misc.health.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:21:50.678 INFO:teuthology.misc.health.smithi097.stderr:last_changed 0.000000 2024-03-25T18:21:50.678 INFO:teuthology.misc.health.smithi097.stderr:created 0.000000 2024-03-25T18:21:50.678 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:21:50.678 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 1 2024-03-25T18:21:50.678 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:21:50.678 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T18:21:50.678 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-c 2024-03-25T18:21:50.678 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:21:50.678 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:50.675+0000 7f665feed640 1 librados: starting msgr at 2024-03-25T18:21:50.678 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:50.675+0000 7f665feed640 1 librados: starting objecter 2024-03-25T18:21:50.678 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:50.675+0000 7f665feed640 1 Processor -- start 2024-03-25T18:21:50.678 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:50.675+0000 7f665feed640 1 -- start start 2024-03-25T18:21:50.678 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:50.675+0000 7f665feed640 1 librados: setting wanted keys 2024-03-25T18:21:50.679 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:50.675+0000 7f665feed640 1 librados: calling monclient init 2024-03-25T18:21:50.679 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:50.675+0000 7f665feed640 10 monclient: init 2024-03-25T18:21:50.679 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:50.676+0000 7f665feed640 10 monclient: _reopen_session rank -1 2024-03-25T18:21:50.679 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:50.676+0000 7f665feed640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:21:50.679 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:50.676+0000 7f665feed640 10 monclient(hunting): picked mon.noname-b con 0x7f66580a98f0 addr v1:172.21.15.98:6789/0 2024-03-25T18:21:50.679 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:50.676+0000 7f665feed640 10 monclient(hunting): picked mon.noname-c con 0x7f665813f710 addr v1:172.21.15.97:6790/0 2024-03-25T18:21:50.679 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:50.676+0000 7f665feed640 10 monclient(hunting): picked mon.noname-a con 0x7f66581430c0 addr v1:172.21.15.97:6789/0 2024-03-25T18:21:50.680 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:50.676+0000 7f665feed640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66581373f0 con 0x7f66581430c0 2024-03-25T18:21:50.680 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:50.676+0000 7f665feed640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66581db280 con 0x7f66580a98f0 2024-03-25T18:21:50.680 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:50.676+0000 7f665feed640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6658138f40 con 0x7f665813f710 2024-03-25T18:21:50.680 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:50.676+0000 7f665feed640 10 monclient(hunting): _renew_subs 2024-03-25T18:21:50.680 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:50.676+0000 7f665feed640 10 monclient(hunting): authenticate will time out at 7922.952150s 2024-03-25T18:21:50.686 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:50.684+0000 7f665e48a640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f66581430c0 0x7f66581332c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:37620/0 (socket says 172.21.15.97:37620) 2024-03-25T18:21:50.686 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:50.684+0000 7f665e48a640 1 -- 172.21.15.97:0/2522694541 learned_addr learned my addr 172.21.15.97:0/2522694541 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:21:50.690 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:50.688+0000 7f664e7fc640 10 client.?.objecter ms_handle_connect 0x7f66580a98f0 2024-03-25T18:21:50.690 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:50.688+0000 7f664e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:50.690 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:50.688+0000 7f664e7fc640 10 client.?.objecter ms_handle_connect 0x7f665813f710 2024-03-25T18:21:50.690 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:50.688+0000 7f664e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:50.690 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:50.688+0000 7f664e7fc640 10 client.?.objecter ms_handle_connect 0x7f66581430c0 2024-03-25T18:21:50.690 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:50.688+0000 7f664e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:50.695 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:50.693+0000 7f664e7fc640 1 -- 172.21.15.97:0/2522694541 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3987569196 0 0) 0x7f66581db280 con 0x7f66580a98f0 2024-03-25T18:21:50.695 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:50.693+0000 7f664e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:21:50.695 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:50.693+0000 7f664e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:50.695 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:50.693+0000 7f664e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:50.695 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:50.693+0000 7f664e7fc640 10 monclient(hunting): my global_id is 4474 2024-03-25T18:21:50.695 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:50.693+0000 7f664e7fc640 1 -- 172.21.15.97:0/2522694541 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f663c004eb0 con 0x7f66580a98f0 2024-03-25T18:21:50.706 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:50.704+0000 7f664e7fc640 1 -- 172.21.15.97:0/2522694541 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2380415905 0 0) 0x7f6658138f40 con 0x7f665813f710 2024-03-25T18:21:50.706 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:50.704+0000 7f664e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:21:50.706 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:50.704+0000 7f664e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:50.706 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:50.704+0000 7f664e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:50.706 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:50.704+0000 7f664e7fc640 10 monclient(hunting): my global_id is 4466 2024-03-25T18:21:50.706 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:50.704+0000 7f664e7fc640 1 -- 172.21.15.97:0/2522694541 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f66581db280 con 0x7f665813f710 2024-03-25T18:21:50.709 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:50.707+0000 7f664e7fc640 1 -- 172.21.15.97:0/2522694541 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1270056255 0 0) 0x7f663c004eb0 con 0x7f66580a98f0 2024-03-25T18:21:50.709 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:50.707+0000 7f664e7fc640 1 -- 172.21.15.97:0/2522694541 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6658138f40 con 0x7f66580a98f0 2024-03-25T18:21:50.709 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:50.707+0000 7f664e7fc640 1 -- 172.21.15.97:0/2522694541 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f6654001870 con 0x7f66580a98f0 2024-03-25T18:21:50.709 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:50.707+0000 7f664e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:50.709 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:50.707+0000 7f664e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:21:50.709 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:50.707+0000 7f664e7fc640 10 monclient(hunting): dump: 2024-03-25T18:21:50.709 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:21:50.709 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:21:50.709 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:21:50.709 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:21:50.709 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:50.709 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:21:50.709 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:21:50.709 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:21:50.709 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:21:50.710 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:21:50.710 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:50.707+0000 7f664e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:21:50.710 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:50.709+0000 7f664e7fc640 1 -- 172.21.15.97:0/2522694541 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1292415870 0 0) 0x7f66581373f0 con 0x7f66581430c0 2024-03-25T18:21:50.710 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:50.709+0000 7f664e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:21:50.710 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:50.709+0000 7f664e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:50.710 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:50.709+0000 7f664e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:50.710 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:50.709+0000 7f664e7fc640 10 monclient(hunting): my global_id is 4470 2024-03-25T18:21:50.711 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:50.709+0000 7f664e7fc640 1 -- 172.21.15.97:0/2522694541 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f663c004eb0 con 0x7f66581430c0 2024-03-25T18:21:50.727 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:50.725+0000 7f664e7fc640 1 -- 172.21.15.97:0/2522694541 <== mon.1 v1:172.21.15.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3339585047 0 0) 0x7f6658138f40 con 0x7f66580a98f0 2024-03-25T18:21:50.727 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:50.725+0000 7f664e7fc640 1 -- 172.21.15.97:0/2522694541 >> v1:172.21.15.97:6790/0 conn(0x7f665813f710 legacy=0x7f6658132bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:50.727 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:50.725+0000 7f664e7fc640 1 -- 172.21.15.97:0/2522694541 >> v1:172.21.15.97:6789/0 conn(0x7f66581430c0 legacy=0x7f66581332c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:50.727 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:50.725+0000 7f664e7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:21:50.727 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:50.725+0000 7f664e7fc640 1 monclient: found mon.b 2024-03-25T18:21:50.727 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:50.725+0000 7f664e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:21:50.727 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:50.725+0000 7f664e7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:21:50.727 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:50.725+0000 7f664e7fc640 1 -- 172.21.15.97:0/2522694541 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66581339d0 con 0x7f66580a98f0 2024-03-25T18:21:50.727 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:50.725+0000 7f664e7fc640 10 monclient: _finish_auth 0 2024-03-25T18:21:50.727 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:50.725+0000 7f664e7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:21:50.727 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:50.725+0000 7f664e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:21:50.727 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:50.725+0000 7f665feed640 5 monclient: authenticate success, global_id 4474 2024-03-25T18:21:50.727 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:50.725+0000 7f665feed640 10 monclient: _renew_subs 2024-03-25T18:21:50.728 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:50.725+0000 7f665feed640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:21:50.728 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:50.725+0000 7f665feed640 1 -- 172.21.15.97:0/2522694541 --> v1:172.21.15.98:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6658137f50 con 0x7f66580a98f0 2024-03-25T18:21:50.728 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:50.725+0000 7f665feed640 10 client.4474.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:21:50.728 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:50.725+0000 7f665feed640 10 monclient: _renew_subs 2024-03-25T18:21:50.728 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:50.725+0000 7f665feed640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:21:50.728 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:50.725+0000 7f665feed640 1 -- 172.21.15.97:0/2522694541 --> v1:172.21.15.98:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f665813c550 con 0x7f66580a98f0 2024-03-25T18:21:50.728 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:50.725+0000 7f665feed640 1 librados: init done 2024-03-25T18:21:50.728 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:50.726+0000 7f665feed640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:50.729 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:50.726+0000 7f665feed640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:50.729 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:50.726+0000 7f665feed640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:21:50.729 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:50.726+0000 7f665feed640 1 -- 172.21.15.97:0/2522694541 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66580b3ef0 con 0x7f66580a98f0 2024-03-25T18:21:51.002 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:51.000+0000 7f664e7fc640 1 -- 172.21.15.97:0/2522694541 <== mon.1 v1:172.21.15.98:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f66540035f0 con 0x7f66580a98f0 2024-03-25T18:21:51.002 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:51.000+0000 7f664e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:21:51.002 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:51.000+0000 7f664e7fc640 1 -- 172.21.15.97:0/2522694541 <== mon.1 v1:172.21.15.98:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f6654001d80 con 0x7f66580a98f0 2024-03-25T18:21:51.002 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:51.000+0000 7f664e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:51.002 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:51.000+0000 7f664e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:21:51.002 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:51.000+0000 7f664e7fc640 10 monclient: dump: 2024-03-25T18:21:51.002 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:21:51.002 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:21:51.002 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:21:51.002 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:21:51.002 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:51.002 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:21:51.003 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:21:51.003 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:21:51.003 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:21:51.003 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:21:51.003 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:51.000+0000 7f664f7fe640 4 set_mon_vals no callback set 2024-03-25T18:21:51.003 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:51.001+0000 7f664e7fc640 1 -- 172.21.15.97:0/2522694541 <== mon.1 v1:172.21.15.98:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2377622137 0 0) 0x7f66540197f0 con 0x7f66580a98f0 2024-03-25T18:21:51.003 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:51.002+0000 7f665d488640 10 monclient: get_auth_request con 0x7f663c06ddb0 auth_method 0 2024-03-25T18:21:51.003 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:51.002+0000 7f664e7fc640 10 client.4474.objecter ms_handle_connect 0x7f663c06ddb0 2024-03-25T18:21:51.007 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:51.006+0000 7f664e7fc640 1 -- 172.21.15.97:0/2522694541 <== mon.1 v1:172.21.15.98:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 430711918 0 0) 0x7f66540535d0 con 0x7f66580a98f0 2024-03-25T18:21:51.007 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:51.006+0000 7f664e7fc640 10 client.4474.objecter ms_dispatch 0x7f6658000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:21:51.007 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:51.006+0000 7f664e7fc640 3 client.4474.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:21:51.008 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:51.006+0000 7f664e7fc640 3 client.4474.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:21:51.008 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:51.006+0000 7f664e7fc640 20 client.4474.objecter dump_active .. 0 homeless 2024-03-25T18:21:51.008 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:51.006+0000 7f664e7fc640 1 -- 172.21.15.97:0/2522694541 <== mon.1 v1:172.21.15.98:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f665404ef10 con 0x7f66580a98f0 2024-03-25T18:21:51.008 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:51.006+0000 7f664e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:51.008 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:51.007+0000 7f664e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:21:51.240 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:51.239+0000 7f665feed640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:51.240 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:51.239+0000 7f665feed640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:51.240 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:51.239+0000 7f665feed640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:21:51.240 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:51.239+0000 7f665feed640 1 -- 172.21.15.97:0/2522694541 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f66580627b0 con 0x7f66580a98f0 2024-03-25T18:21:51.322 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:51.320+0000 7f664e7fc640 1 -- 172.21.15.97:0/2522694541 <== mon.1 v1:172.21.15.98:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2169427412) 0x7f6654051010 con 0x7f66580a98f0 2024-03-25T18:21:51.322 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:51.320+0000 7f664e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:51.322 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:51.320+0000 7f664e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:21:51.322 INFO:teuthology.misc.health.smithi097.stdout: 2024-03-25T18:21:51.323 INFO:teuthology.misc.health.smithi097.stdout:{"epoch":10,"fsid":"4c3d472c-79b8-486b-9e3d-2738d4306754","created":"2024-03-25T18:16:14.737309+0000","modified":"2024-03-25T18:16:20.357492+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:20.357492+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cb5152b9-262f-45c3-94b8-18d0e81df558","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6daf8793-0838-473d-a7d6-9a0d2894c75d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"136f1493-3b9c-4f43-a2e7-641482d69dae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"397d538c-4f64-4daf-9e8b-662d09f91327","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4f1df08-663f-4bda-93f8-3878ab7c24b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c4b18d70-a3ce-4b15-9b70-eae5525bd761","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8dd4033a-de0e-4f8a-96dc-59f3c1f825b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0077722e-ffa6-4a3d-9891-929d9869b286","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:21:51.325 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:51.323+0000 7f665feed640 10 librados: watch_flush enter 2024-03-25T18:21:51.325 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:51.323+0000 7f665feed640 10 librados: watch_flush exit 2024-03-25T18:21:51.325 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:51.323+0000 7f665feed640 20 client.4474.objecter shutdown clearing up homeless session... 2024-03-25T18:21:51.325 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:51.323+0000 7f665feed640 10 client.4474.objecter successfully canceled tick 2024-03-25T18:21:51.325 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:51.323+0000 7f665feed640 1 -- 172.21.15.97:0/2522694541 >> v1:172.21.15.98:6800/800387573 conn(0x7f663c06ddb0 legacy=0x7f663c070270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:51.325 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:51.323+0000 7f665feed640 10 monclient: shutdown 2024-03-25T18:21:51.325 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:51.323+0000 7f665feed640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:21:51.325 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:51.323+0000 7f665feed640 1 -- 172.21.15.97:0/2522694541 >> v1:172.21.15.98:6789/0 conn(0x7f66580a98f0 legacy=0x7f66581dab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:51.325 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:51.323+0000 7f665feed640 1 -- 172.21.15.97:0/2522694541 shutdown_connections 2024-03-25T18:21:51.326 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:51.323+0000 7f665feed640 1 -- 172.21.15.97:0/2522694541 >> 172.21.15.97:0/2522694541 conn(0x7f66580b7ea0 msgr2=0x7f6658074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:21:51.326 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:51.323+0000 7f665feed640 1 -- 172.21.15.97:0/2522694541 shutdown_connections 2024-03-25T18:21:51.326 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:51.323+0000 7f665feed640 1 -- 172.21.15.97:0/2522694541 wait complete. 2024-03-25T18:21:51.326 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:51.324+0000 7f665feed640 1 librados: shutdown 2024-03-25T18:21:51.337 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:21:52.865 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:05:54.903 34931== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:52.914 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:05:54.953 34931== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:52.951 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:05:54.990 34931== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:52.961 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:05:55.004 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:52.966 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:05:55.004 34935== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:52.973 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:05:55.012 34935== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:52.974 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:05:55.017 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:52.996 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:05:55.040 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:53.016 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:05:55.055 34935== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:58.143 DEBUG:teuthology.orchestra.run.smithi097:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:21:58.258 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.257+0000 7fdcb3972640 10 monclient: get_monmap_and_config 2024-03-25T18:21:58.259 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.257+0000 7fdcb3972640 10 monclient: build_initial_monmap 2024-03-25T18:21:58.259 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.257+0000 7fdcb3972640 10 monclient: monmap: 2024-03-25T18:21:58.259 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:21:58.259 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:21:58.259 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:21:58.258794+0000 2024-03-25T18:21:58.259 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:21:58.258794+0000 2024-03-25T18:21:58.259 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:21:58.259 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:21:58.259 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:21:58.259 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T18:21:58.259 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-b 2024-03-25T18:21:58.259 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:21:58.261 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.258+0000 7fdcb3972640 1 Processor -- start 2024-03-25T18:21:58.261 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.258+0000 7fdcb3972640 1 -- start start 2024-03-25T18:21:58.261 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.258+0000 7fdcb3972640 10 monclient: init 2024-03-25T18:21:58.261 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.258+0000 7fdcb3972640 10 monclient: _reopen_session rank -1 2024-03-25T18:21:58.261 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.258+0000 7fdcb3972640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:21:58.261 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.259+0000 7fdcb3972640 10 monclient(hunting): picked mon.noname-b con 0x7fdcac0a98f0 addr v1:172.21.15.97:6790/0 2024-03-25T18:21:58.261 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.259+0000 7fdcb3972640 10 monclient(hunting): picked mon.noname-a con 0x7fdcac13f710 addr v1:172.21.15.97:6789/0 2024-03-25T18:21:58.261 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.259+0000 7fdcb3972640 10 monclient(hunting): picked mon.noname-c con 0x7fdcac1430c0 addr v1:172.21.15.98:6789/0 2024-03-25T18:21:58.261 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.259+0000 7fdcb3972640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcac142520 con 0x7fdcac13f710 2024-03-25T18:21:58.261 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.259+0000 7fdcb3972640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcac0b28a0 con 0x7fdcac1430c0 2024-03-25T18:21:58.261 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.259+0000 7fdcb3972640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcac0b2a30 con 0x7fdcac0a98f0 2024-03-25T18:21:58.262 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.259+0000 7fdcb3972640 10 monclient(hunting): _renew_subs 2024-03-25T18:21:58.262 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.259+0000 7fdcb3972640 10 monclient(hunting): authenticate will time out at 7930.535053s 2024-03-25T18:21:58.262 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.259+0000 7fdcb1f0f640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fdcac1430c0 0x7fdcac14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.97:45998/0 (socket says 172.21.15.97:45998) 2024-03-25T18:21:58.262 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.259+0000 7fdcb1f0f640 1 -- 172.21.15.97:0/2120501802 learned_addr learned my addr 172.21.15.97:0/2120501802 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:21:58.338 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.336+0000 7fdca3fff640 1 -- 172.21.15.97:0/2120501802 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1651539651 0 0) 0x7fdcac0b28a0 con 0x7fdcac1430c0 2024-03-25T18:21:58.338 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.336+0000 7fdca3fff640 10 monclient(hunting): _negotiate 2024-03-25T18:21:58.338 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.336+0000 7fdca3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:58.338 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.336+0000 7fdca3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:58.338 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.336+0000 7fdca3fff640 10 monclient(hunting): my global_id is 4477 2024-03-25T18:21:58.338 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.337+0000 7fdca3fff640 1 -- 172.21.15.97:0/2120501802 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc98003520 con 0x7fdcac1430c0 2024-03-25T18:21:58.340 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.339+0000 7fdca3fff640 1 -- 172.21.15.97:0/2120501802 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1140122093 0 0) 0x7fdcac142520 con 0x7fdcac13f710 2024-03-25T18:21:58.341 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.339+0000 7fdca3fff640 10 monclient(hunting): _negotiate 2024-03-25T18:21:58.341 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.339+0000 7fdca3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:58.341 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.339+0000 7fdca3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:58.341 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.339+0000 7fdca3fff640 10 monclient(hunting): my global_id is 4473 2024-03-25T18:21:58.341 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.339+0000 7fdca3fff640 1 -- 172.21.15.97:0/2120501802 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdcac0b28a0 con 0x7fdcac13f710 2024-03-25T18:21:58.342 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.340+0000 7fdca3fff640 1 -- 172.21.15.97:0/2120501802 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1957512249 0 0) 0x7fdcac0b2a30 con 0x7fdcac0a98f0 2024-03-25T18:21:58.342 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.340+0000 7fdca3fff640 10 monclient(hunting): _negotiate 2024-03-25T18:21:58.342 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.340+0000 7fdca3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:58.342 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.340+0000 7fdca3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:58.342 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.340+0000 7fdca3fff640 10 monclient(hunting): my global_id is 4469 2024-03-25T18:21:58.342 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.340+0000 7fdca3fff640 1 -- 172.21.15.97:0/2120501802 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdcac142520 con 0x7fdcac0a98f0 2024-03-25T18:21:58.351 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.349+0000 7fdca3fff640 1 -- 172.21.15.97:0/2120501802 <== mon.2 v1:172.21.15.97:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2501150005 0 0) 0x7fdcac142520 con 0x7fdcac0a98f0 2024-03-25T18:21:58.351 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.349+0000 7fdca3fff640 1 -- 172.21.15.97:0/2120501802 >> v1:172.21.15.98:6789/0 conn(0x7fdcac1430c0 legacy=0x7fdcac14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:58.351 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.349+0000 7fdca3fff640 1 -- 172.21.15.97:0/2120501802 >> v1:172.21.15.97:6789/0 conn(0x7fdcac13f710 legacy=0x7fdcac1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:58.351 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.349+0000 7fdca3fff640 10 monclient: _finish_hunting 0 2024-03-25T18:21:58.351 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.349+0000 7fdca3fff640 1 monclient: found mon.noname-b 2024-03-25T18:21:58.351 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.349+0000 7fdca3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:21:58.351 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.349+0000 7fdca3fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.97:6790/0 2024-03-25T18:21:58.351 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.349+0000 7fdca3fff640 1 -- 172.21.15.97:0/2120501802 --> v1:172.21.15.97:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcac0b44c0 con 0x7fdcac0a98f0 2024-03-25T18:21:58.352 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.349+0000 7fdca3fff640 10 monclient: _finish_auth 0 2024-03-25T18:21:58.352 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.349+0000 7fdca3fff640 10 monclient: _check_auth_tickets 2024-03-25T18:21:58.352 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.349+0000 7fdca3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:21:58.352 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.349+0000 7fdca3fff640 1 -- 172.21.15.97:0/2120501802 <== mon.2 v1:172.21.15.97:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7fdca80015b0 con 0x7fdcac0a98f0 2024-03-25T18:21:58.352 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.349+0000 7fdca3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:58.352 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.349+0000 7fdca3fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:21:58.352 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.349+0000 7fdca3fff640 10 monclient: dump: 2024-03-25T18:21:58.352 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:21:58.352 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:21:58.352 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:21:58.352 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:21:58.352 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:58.352 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:21:58.352 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:21:58.353 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:21:58.353 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:21:58.353 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:21:58.353 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.349+0000 7fdcb3972640 5 monclient: authenticate success, global_id 4469 2024-03-25T18:21:58.353 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.349+0000 7fdcb3972640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:21:58.365 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.364+0000 7fdca3fff640 1 -- 172.21.15.97:0/2120501802 <== mon.2 v1:172.21.15.97:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdca8001710 con 0x7fdcac0a98f0 2024-03-25T18:21:58.365 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.364+0000 7fdca3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:21:58.365 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.364+0000 7fdca3fff640 1 -- 172.21.15.97:0/2120501802 <== mon.2 v1:172.21.15.97:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7fdca8001b60 con 0x7fdcac0a98f0 2024-03-25T18:21:58.365 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.364+0000 7fdca3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:58.365 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.364+0000 7fdca3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:21:58.366 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.364+0000 7fdca3fff640 10 monclient: dump: 2024-03-25T18:21:58.366 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:21:58.366 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:21:58.366 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:21:58.366 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:21:58.366 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:58.366 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:21:58.366 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:21:58.366 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:21:58.366 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:21:58.366 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:21:58.366 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.364+0000 7fdcb3972640 10 monclient: get_monmap_and_config success 2024-03-25T18:21:58.366 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.364+0000 7fdcb3972640 4 set_mon_vals no callback set 2024-03-25T18:21:58.368 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.364+0000 7fdcb3972640 10 monclient: shutdown 2024-03-25T18:21:58.369 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.365+0000 7fdcb3972640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:21:58.369 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.365+0000 7fdcb3972640 1 -- 172.21.15.97:0/2120501802 >> v1:172.21.15.97:6790/0 conn(0x7fdcac0a98f0 legacy=0x7fdcac0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:58.369 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.365+0000 7fdcb3972640 1 -- 172.21.15.97:0/2120501802 shutdown_connections 2024-03-25T18:21:58.369 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.365+0000 7fdcb3972640 1 -- 172.21.15.97:0/2120501802 >> 172.21.15.97:0/2120501802 conn(0x7fdcac0b7ea0 msgr2=0x7fdcac0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:21:58.369 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.365+0000 7fdcb3972640 1 -- 172.21.15.97:0/2120501802 shutdown_connections 2024-03-25T18:21:58.369 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.365+0000 7fdcb3972640 1 -- 172.21.15.97:0/2120501802 wait complete. 2024-03-25T18:21:58.369 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.365+0000 7fdcb3972640 10 monclient: build_initial_monmap 2024-03-25T18:21:58.369 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.365+0000 7fdcb3972640 10 monclient: monmap: 2024-03-25T18:21:58.369 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:21:58.369 INFO:teuthology.misc.health.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:21:58.369 INFO:teuthology.misc.health.smithi097.stderr:last_changed 0.000000 2024-03-25T18:21:58.369 INFO:teuthology.misc.health.smithi097.stderr:created 0.000000 2024-03-25T18:21:58.370 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:21:58.370 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 1 2024-03-25T18:21:58.370 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:21:58.370 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T18:21:58.370 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-c 2024-03-25T18:21:58.370 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:21:58.370 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.366+0000 7fdcb3972640 1 librados: starting msgr at 2024-03-25T18:21:58.370 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.366+0000 7fdcb3972640 1 librados: starting objecter 2024-03-25T18:21:58.370 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.366+0000 7fdcb3972640 1 Processor -- start 2024-03-25T18:21:58.370 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.366+0000 7fdcb3972640 1 -- start start 2024-03-25T18:21:58.370 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.366+0000 7fdcb3972640 1 librados: setting wanted keys 2024-03-25T18:21:58.370 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.366+0000 7fdcb3972640 1 librados: calling monclient init 2024-03-25T18:21:58.371 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.366+0000 7fdcb3972640 10 monclient: init 2024-03-25T18:21:58.371 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.367+0000 7fdcb3972640 10 monclient: _reopen_session rank -1 2024-03-25T18:21:58.371 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.367+0000 7fdcb3972640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:21:58.371 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.367+0000 7fdcb3972640 10 monclient(hunting): picked mon.noname-b con 0x7fdcac0a98f0 addr v1:172.21.15.98:6789/0 2024-03-25T18:21:58.371 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.367+0000 7fdcb3972640 10 monclient(hunting): picked mon.noname-a con 0x7fdcac13f710 addr v1:172.21.15.97:6789/0 2024-03-25T18:21:58.371 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.367+0000 7fdcb3972640 10 monclient(hunting): picked mon.noname-c con 0x7fdcac1430c0 addr v1:172.21.15.97:6790/0 2024-03-25T18:21:58.371 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.367+0000 7fdcb3972640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcac138050 con 0x7fdcac13f710 2024-03-25T18:21:58.371 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.367+0000 7fdcb3972640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcac137d70 con 0x7fdcac0a98f0 2024-03-25T18:21:58.371 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.367+0000 7fdcb3972640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcac138f20 con 0x7fdcac1430c0 2024-03-25T18:21:58.371 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.367+0000 7fdcb3972640 10 monclient(hunting): _renew_subs 2024-03-25T18:21:58.371 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.367+0000 7fdcb3972640 10 monclient(hunting): authenticate will time out at 7930.643147s 2024-03-25T18:21:58.379 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.378+0000 7fdcb0f0d640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fdcac13f710 0x7fdcac138250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:42390/0 (socket says 172.21.15.97:42390) 2024-03-25T18:21:58.379 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.378+0000 7fdcb0f0d640 1 -- 172.21.15.97:0/1582446446 learned_addr learned my addr 172.21.15.97:0/1582446446 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:21:58.383 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.382+0000 7fdca1ffb640 10 client.?.objecter ms_handle_connect 0x7fdcac13f710 2024-03-25T18:21:58.384 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.382+0000 7fdca1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:58.388 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.387+0000 7fdca1ffb640 10 client.?.objecter ms_handle_connect 0x7fdcac1430c0 2024-03-25T18:21:58.389 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.387+0000 7fdca1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:58.389 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.388+0000 7fdca1ffb640 10 client.?.objecter ms_handle_connect 0x7fdcac0a98f0 2024-03-25T18:21:58.389 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.388+0000 7fdca1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:58.402 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.400+0000 7fdca1ffb640 1 -- 172.21.15.97:0/1582446446 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1837163795 0 0) 0x7fdcac137d70 con 0x7fdcac0a98f0 2024-03-25T18:21:58.402 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.400+0000 7fdca1ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:21:58.402 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.400+0000 7fdca1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:58.402 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.400+0000 7fdca1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:58.402 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.400+0000 7fdca1ffb640 10 monclient(hunting): my global_id is 4480 2024-03-25T18:21:58.402 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.400+0000 7fdca1ffb640 1 -- 172.21.15.97:0/1582446446 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc90004eb0 con 0x7fdcac0a98f0 2024-03-25T18:21:58.411 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.409+0000 7fdca1ffb640 1 -- 172.21.15.97:0/1582446446 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3392216992 0 0) 0x7fdcac138050 con 0x7fdcac13f710 2024-03-25T18:21:58.411 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.409+0000 7fdca1ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:21:58.411 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.409+0000 7fdca1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:58.411 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.409+0000 7fdca1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:58.411 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.409+0000 7fdca1ffb640 10 monclient(hunting): my global_id is 4476 2024-03-25T18:21:58.411 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.409+0000 7fdca1ffb640 1 -- 172.21.15.97:0/1582446446 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdcac137d70 con 0x7fdcac13f710 2024-03-25T18:21:58.416 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.414+0000 7fdca1ffb640 1 -- 172.21.15.97:0/1582446446 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2189113572 0 0) 0x7fdcac138f20 con 0x7fdcac1430c0 2024-03-25T18:21:58.416 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.414+0000 7fdca1ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:21:58.416 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.414+0000 7fdca1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:58.416 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.414+0000 7fdca1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:58.416 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.414+0000 7fdca1ffb640 10 monclient(hunting): my global_id is 4472 2024-03-25T18:21:58.416 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.414+0000 7fdca1ffb640 1 -- 172.21.15.97:0/1582446446 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdcac138050 con 0x7fdcac1430c0 2024-03-25T18:21:58.426 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.424+0000 7fdca1ffb640 1 -- 172.21.15.97:0/1582446446 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1863038843 0 0) 0x7fdcac137d70 con 0x7fdcac13f710 2024-03-25T18:21:58.426 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.424+0000 7fdca1ffb640 1 -- 172.21.15.97:0/1582446446 --> v1:172.21.15.97:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdcac138f20 con 0x7fdcac13f710 2024-03-25T18:21:58.426 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.424+0000 7fdca1ffb640 1 -- 172.21.15.97:0/1582446446 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7fdc9c0019f0 con 0x7fdcac13f710 2024-03-25T18:21:58.426 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.424+0000 7fdca1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:58.426 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.424+0000 7fdca1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:21:58.426 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.424+0000 7fdca1ffb640 10 monclient(hunting): dump: 2024-03-25T18:21:58.426 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:21:58.426 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:21:58.427 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:21:58.427 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:21:58.427 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:58.427 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:21:58.427 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:21:58.427 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:21:58.427 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:21:58.427 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:21:58.427 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.424+0000 7fdca1ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:21:58.431 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.429+0000 7fdca1ffb640 1 -- 172.21.15.97:0/1582446446 <== mon.0 v1:172.21.15.97:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3191837297 0 0) 0x7fdcac138f20 con 0x7fdcac13f710 2024-03-25T18:21:58.431 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.429+0000 7fdca1ffb640 1 -- 172.21.15.97:0/1582446446 >> v1:172.21.15.97:6790/0 conn(0x7fdcac1430c0 legacy=0x7fdcac1e1bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:58.431 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.429+0000 7fdca1ffb640 1 -- 172.21.15.97:0/1582446446 >> v1:172.21.15.98:6789/0 conn(0x7fdcac0a98f0 legacy=0x7fdcac1db100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:58.431 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.429+0000 7fdca1ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:21:58.431 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.429+0000 7fdca1ffb640 1 monclient: found mon.a 2024-03-25T18:21:58.431 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.429+0000 7fdca1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:21:58.431 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.429+0000 7fdca1ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:21:58.431 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.429+0000 7fdca1ffb640 1 -- 172.21.15.97:0/1582446446 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcac1379b0 con 0x7fdcac13f710 2024-03-25T18:21:58.431 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.429+0000 7fdca1ffb640 10 monclient: _finish_auth 0 2024-03-25T18:21:58.431 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.429+0000 7fdca1ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:21:58.431 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.429+0000 7fdca1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:21:58.432 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.429+0000 7fdcb3972640 5 monclient: authenticate success, global_id 4476 2024-03-25T18:21:58.432 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.429+0000 7fdcb3972640 10 monclient: _renew_subs 2024-03-25T18:21:58.432 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.429+0000 7fdcb3972640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:21:58.432 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.429+0000 7fdcb3972640 1 -- 172.21.15.97:0/1582446446 --> v1:172.21.15.97:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcac138960 con 0x7fdcac13f710 2024-03-25T18:21:58.432 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.429+0000 7fdcb3972640 10 client.4476.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:21:58.432 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.429+0000 7fdcb3972640 10 monclient: _renew_subs 2024-03-25T18:21:58.432 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.429+0000 7fdcb3972640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:21:58.432 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.429+0000 7fdcb3972640 1 -- 172.21.15.97:0/1582446446 --> v1:172.21.15.97:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcac138d00 con 0x7fdcac13f710 2024-03-25T18:21:58.432 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.429+0000 7fdcb3972640 1 librados: init done 2024-03-25T18:21:58.433 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.430+0000 7fdcb3972640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:58.433 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.430+0000 7fdcb3972640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:58.433 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.430+0000 7fdcb3972640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:21:58.433 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.430+0000 7fdcb3972640 1 -- 172.21.15.97:0/1582446446 --> v1:172.21.15.97:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdcac137ba0 con 0x7fdcac13f710 2024-03-25T18:21:58.746 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.744+0000 7fdca1ffb640 1 -- 172.21.15.97:0/1582446446 <== mon.0 v1:172.21.15.97:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdc9c0035f0 con 0x7fdcac13f710 2024-03-25T18:21:58.746 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.744+0000 7fdca1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:21:58.746 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.744+0000 7fdca1ffb640 1 -- 172.21.15.97:0/1582446446 <== mon.0 v1:172.21.15.97:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7fdc9c003a00 con 0x7fdcac13f710 2024-03-25T18:21:58.746 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.744+0000 7fdca1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:58.746 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.744+0000 7fdca1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:21:58.746 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.744+0000 7fdca1ffb640 10 monclient: dump: 2024-03-25T18:21:58.746 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:21:58.747 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:21:58.747 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:21:58.747 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:21:58.747 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:58.747 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:21:58.747 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:21:58.747 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:21:58.747 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:21:58.747 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:21:58.747 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.744+0000 7fdca37fe640 4 set_mon_vals no callback set 2024-03-25T18:21:58.748 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.745+0000 7fdca1ffb640 1 -- 172.21.15.97:0/1582446446 <== mon.0 v1:172.21.15.97:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2377622137 0 0) 0x7fdc9c003c20 con 0x7fdcac13f710 2024-03-25T18:21:58.748 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.746+0000 7fdcb170e640 10 monclient: get_auth_request con 0x7fdc9006de20 auth_method 0 2024-03-25T18:21:58.748 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.746+0000 7fdca1ffb640 10 client.4476.objecter ms_handle_connect 0x7fdc9006de20 2024-03-25T18:21:58.754 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.752+0000 7fdca1ffb640 1 -- 172.21.15.97:0/1582446446 <== mon.0 v1:172.21.15.97:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 430711918 0 0) 0x7fdc9c054810 con 0x7fdcac13f710 2024-03-25T18:21:58.754 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.752+0000 7fdca1ffb640 10 client.4476.objecter ms_dispatch 0x7fdcac000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:21:58.754 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.752+0000 7fdca1ffb640 3 client.4476.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:21:58.755 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.752+0000 7fdca1ffb640 3 client.4476.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:21:58.755 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.752+0000 7fdca1ffb640 20 client.4476.objecter dump_active .. 0 homeless 2024-03-25T18:21:58.755 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.752+0000 7fdca1ffb640 1 -- 172.21.15.97:0/1582446446 <== mon.0 v1:172.21.15.97:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdc9c050320 con 0x7fdcac13f710 2024-03-25T18:21:58.755 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.752+0000 7fdca1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:58.755 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:58.752+0000 7fdca1ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:21:59.006 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:59.005+0000 7fdcb3972640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:59.007 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:59.005+0000 7fdcb3972640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:59.007 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:59.005+0000 7fdcb3972640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:21:59.007 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:59.005+0000 7fdcb3972640 1 -- 172.21.15.97:0/1582446446 --> v1:172.21.15.97:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdcac138af0 con 0x7fdcac13f710 2024-03-25T18:21:59.035 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:59.033+0000 7fdca1ffb640 1 -- 172.21.15.97:0/1582446446 <== mon.0 v1:172.21.15.97:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2169427412) 0x7fdc9c052250 con 0x7fdcac13f710 2024-03-25T18:21:59.035 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:59.033+0000 7fdca1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:59.035 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:59.033+0000 7fdca1ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:21:59.035 INFO:teuthology.misc.health.smithi097.stdout: 2024-03-25T18:21:59.035 INFO:teuthology.misc.health.smithi097.stdout:{"epoch":10,"fsid":"4c3d472c-79b8-486b-9e3d-2738d4306754","created":"2024-03-25T18:16:14.737309+0000","modified":"2024-03-25T18:16:20.357492+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:20.357492+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cb5152b9-262f-45c3-94b8-18d0e81df558","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6daf8793-0838-473d-a7d6-9a0d2894c75d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"136f1493-3b9c-4f43-a2e7-641482d69dae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"397d538c-4f64-4daf-9e8b-662d09f91327","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4f1df08-663f-4bda-93f8-3878ab7c24b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c4b18d70-a3ce-4b15-9b70-eae5525bd761","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8dd4033a-de0e-4f8a-96dc-59f3c1f825b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0077722e-ffa6-4a3d-9891-929d9869b286","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:21:59.038 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:59.036+0000 7fdcb3972640 10 librados: watch_flush enter 2024-03-25T18:21:59.038 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:59.036+0000 7fdcb3972640 10 librados: watch_flush exit 2024-03-25T18:21:59.038 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:59.036+0000 7fdcb3972640 20 client.4476.objecter shutdown clearing up homeless session... 2024-03-25T18:21:59.038 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:59.036+0000 7fdcb3972640 10 client.4476.objecter successfully canceled tick 2024-03-25T18:21:59.038 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:59.036+0000 7fdcb3972640 1 -- 172.21.15.97:0/1582446446 >> v1:172.21.15.98:6800/800387573 conn(0x7fdc9006de20 legacy=0x7fdc900702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:59.038 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:59.036+0000 7fdcb3972640 10 monclient: shutdown 2024-03-25T18:21:59.038 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:59.036+0000 7fdcb3972640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:21:59.038 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:59.036+0000 7fdcb3972640 1 -- 172.21.15.97:0/1582446446 >> v1:172.21.15.97:6789/0 conn(0x7fdcac13f710 legacy=0x7fdcac138250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:59.039 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:59.036+0000 7fdcb3972640 1 -- 172.21.15.97:0/1582446446 shutdown_connections 2024-03-25T18:21:59.039 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:59.036+0000 7fdcb3972640 1 -- 172.21.15.97:0/1582446446 >> 172.21.15.97:0/1582446446 conn(0x7fdcac0b7ea0 msgr2=0x7fdcac074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:21:59.039 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:59.036+0000 7fdcb3972640 1 -- 172.21.15.97:0/1582446446 shutdown_connections 2024-03-25T18:21:59.039 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:59.036+0000 7fdcb3972640 1 -- 172.21.15.97:0/1582446446 wait complete. 2024-03-25T18:21:59.039 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:21:59.036+0000 7fdcb3972640 1 librados: shutdown 2024-03-25T18:21:59.050 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:22:05.857 DEBUG:teuthology.orchestra.run.smithi097:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:22:05.963 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:05.961+0000 7f9815a05640 10 monclient: get_monmap_and_config 2024-03-25T18:22:05.963 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:05.961+0000 7f9815a05640 10 monclient: build_initial_monmap 2024-03-25T18:22:05.963 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:05.961+0000 7f9815a05640 10 monclient: monmap: 2024-03-25T18:22:05.963 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:22:05.963 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:22:05.963 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:22:05.962570+0000 2024-03-25T18:22:05.963 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:22:05.962570+0000 2024-03-25T18:22:05.963 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:22:05.963 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:22:05.963 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:22:05.963 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T18:22:05.964 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-b 2024-03-25T18:22:05.964 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:22:05.964 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:05.961+0000 7f9815a05640 1 Processor -- start 2024-03-25T18:22:05.964 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:05.961+0000 7f9815a05640 1 -- start start 2024-03-25T18:22:05.964 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:05.961+0000 7f9815a05640 10 monclient: init 2024-03-25T18:22:05.965 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:05.962+0000 7f9815a05640 10 monclient: _reopen_session rank -1 2024-03-25T18:22:05.965 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:05.962+0000 7f9815a05640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:22:05.965 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:05.962+0000 7f9815a05640 10 monclient(hunting): picked mon.noname-c con 0x7f98100b4f40 addr v1:172.21.15.98:6789/0 2024-03-25T18:22:05.965 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:05.962+0000 7f9815a05640 10 monclient(hunting): picked mon.noname-a con 0x7f981013fd90 addr v1:172.21.15.97:6789/0 2024-03-25T18:22:05.965 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:05.962+0000 7f9815a05640 10 monclient(hunting): picked mon.noname-b con 0x7f98100b4540 addr v1:172.21.15.97:6790/0 2024-03-25T18:22:05.965 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:05.962+0000 7f9815a05640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98100aa0c0 con 0x7f981013fd90 2024-03-25T18:22:05.965 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:05.962+0000 7f9815a05640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98100ab170 con 0x7f98100b4f40 2024-03-25T18:22:05.965 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:05.962+0000 7f9815a05640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98100a9ba0 con 0x7f98100b4540 2024-03-25T18:22:05.965 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:05.962+0000 7f9815a05640 10 monclient(hunting): _renew_subs 2024-03-25T18:22:05.965 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:05.962+0000 7f9815a05640 10 monclient(hunting): authenticate will time out at 7938.238512s 2024-03-25T18:22:06.000 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:05.999+0000 7f980effd640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f98100b4f40 0x7f981013f680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.97:33998/0 (socket says 172.21.15.97:33998) 2024-03-25T18:22:06.000 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:05.999+0000 7f980effd640 1 -- 172.21.15.97:0/14103772 learned_addr learned my addr 172.21.15.97:0/14103772 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:22:06.018 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.017+0000 7f980dffb640 1 -- 172.21.15.97:0/14103772 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3701469241 0 0) 0x7f98100aa0c0 con 0x7f981013fd90 2024-03-25T18:22:06.018 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.017+0000 7f980dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:22:06.018 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.017+0000 7f980dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:06.018 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.017+0000 7f980dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:06.019 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.017+0000 7f980dffb640 10 monclient(hunting): my global_id is 4479 2024-03-25T18:22:06.019 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.017+0000 7f980dffb640 1 -- 172.21.15.97:0/14103772 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97f8003520 con 0x7f981013fd90 2024-03-25T18:22:06.019 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.018+0000 7f980dffb640 1 -- 172.21.15.97:0/14103772 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2669594987 0 0) 0x7f98100ab170 con 0x7f98100b4f40 2024-03-25T18:22:06.020 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.018+0000 7f980dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:22:06.020 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.018+0000 7f980dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:06.020 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.018+0000 7f980dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:06.020 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.018+0000 7f980dffb640 10 monclient(hunting): my global_id is 4483 2024-03-25T18:22:06.020 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.018+0000 7f980dffb640 1 -- 172.21.15.97:0/14103772 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98100aa0c0 con 0x7f98100b4f40 2024-03-25T18:22:06.022 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.020+0000 7f980dffb640 1 -- 172.21.15.97:0/14103772 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1706720352 0 0) 0x7f98100a9ba0 con 0x7f98100b4540 2024-03-25T18:22:06.022 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.021+0000 7f980dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:22:06.022 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.021+0000 7f980dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:06.022 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.021+0000 7f980dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:06.022 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.021+0000 7f980dffb640 10 monclient(hunting): my global_id is 4475 2024-03-25T18:22:06.022 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.021+0000 7f980dffb640 1 -- 172.21.15.97:0/14103772 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98100ab170 con 0x7f98100b4540 2024-03-25T18:22:06.029 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.028+0000 7f980dffb640 1 -- 172.21.15.97:0/14103772 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2440182615 0 0) 0x7f97f8003520 con 0x7f981013fd90 2024-03-25T18:22:06.029 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.028+0000 7f980dffb640 1 -- 172.21.15.97:0/14103772 >> v1:172.21.15.97:6790/0 conn(0x7f98100b4540 legacy=0x7f981014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:06.029 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.028+0000 7f980dffb640 1 -- 172.21.15.97:0/14103772 >> v1:172.21.15.98:6789/0 conn(0x7f98100b4f40 legacy=0x7f981013f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:06.029 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.028+0000 7f980dffb640 10 monclient: _finish_hunting 0 2024-03-25T18:22:06.030 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.028+0000 7f980dffb640 1 monclient: found mon.noname-a 2024-03-25T18:22:06.030 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.028+0000 7f980dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:22:06.030 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.028+0000 7f980dffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.97:6789/0 2024-03-25T18:22:06.030 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.028+0000 7f980dffb640 1 -- 172.21.15.97:0/14103772 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98100a9680 con 0x7f981013fd90 2024-03-25T18:22:06.030 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.028+0000 7f980dffb640 10 monclient: _finish_auth 0 2024-03-25T18:22:06.030 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.028+0000 7f980dffb640 10 monclient: _check_auth_tickets 2024-03-25T18:22:06.030 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.028+0000 7f980dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:22:06.030 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.028+0000 7f980dffb640 1 -- 172.21.15.97:0/14103772 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f9804001690 con 0x7f981013fd90 2024-03-25T18:22:06.030 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.028+0000 7f980dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:06.030 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.028+0000 7f980dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:22:06.030 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.028+0000 7f980dffb640 10 monclient: dump: 2024-03-25T18:22:06.030 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:22:06.030 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:22:06.030 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:22:06.030 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:22:06.031 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:06.031 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:22:06.031 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:22:06.031 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:22:06.031 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:22:06.031 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:22:06.031 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.028+0000 7f9815a05640 5 monclient: authenticate success, global_id 4479 2024-03-25T18:22:06.031 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.028+0000 7f9815a05640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:22:06.051 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.050+0000 7f980dffb640 1 -- 172.21.15.97:0/14103772 <== mon.0 v1:172.21.15.97:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f98040017f0 con 0x7f981013fd90 2024-03-25T18:22:06.051 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.050+0000 7f980dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:22:06.052 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.050+0000 7f980dffb640 1 -- 172.21.15.97:0/14103772 <== mon.0 v1:172.21.15.97:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f9804001c00 con 0x7f981013fd90 2024-03-25T18:22:06.052 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.050+0000 7f980dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:06.052 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.050+0000 7f980dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:22:06.052 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.050+0000 7f980dffb640 10 monclient: dump: 2024-03-25T18:22:06.052 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:22:06.052 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:22:06.052 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:22:06.052 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:22:06.052 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:06.052 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:22:06.052 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:22:06.052 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:22:06.052 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:22:06.052 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:22:06.052 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.050+0000 7f9815a05640 10 monclient: get_monmap_and_config success 2024-03-25T18:22:06.053 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.050+0000 7f9815a05640 4 set_mon_vals no callback set 2024-03-25T18:22:06.054 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.051+0000 7f9815a05640 10 monclient: shutdown 2024-03-25T18:22:06.054 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.051+0000 7f9815a05640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:22:06.054 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.051+0000 7f9815a05640 1 -- 172.21.15.97:0/14103772 >> v1:172.21.15.97:6789/0 conn(0x7f981013fd90 legacy=0x7f98101423f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:06.054 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.051+0000 7f9815a05640 1 -- 172.21.15.97:0/14103772 shutdown_connections 2024-03-25T18:22:06.054 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.051+0000 7f9815a05640 1 -- 172.21.15.97:0/14103772 >> 172.21.15.97:0/14103772 conn(0x7f98100617a0 msgr2=0x7f9810061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:22:06.054 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.051+0000 7f9815a05640 1 -- 172.21.15.97:0/14103772 shutdown_connections 2024-03-25T18:22:06.054 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.051+0000 7f9815a05640 1 -- 172.21.15.97:0/14103772 wait complete. 2024-03-25T18:22:06.054 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.052+0000 7f9815a05640 10 monclient: build_initial_monmap 2024-03-25T18:22:06.054 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.052+0000 7f9815a05640 10 monclient: monmap: 2024-03-25T18:22:06.054 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:22:06.054 INFO:teuthology.misc.health.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:22:06.055 INFO:teuthology.misc.health.smithi097.stderr:last_changed 0.000000 2024-03-25T18:22:06.055 INFO:teuthology.misc.health.smithi097.stderr:created 0.000000 2024-03-25T18:22:06.055 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:22:06.055 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 1 2024-03-25T18:22:06.055 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:22:06.055 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T18:22:06.055 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-c 2024-03-25T18:22:06.055 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:22:06.055 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.052+0000 7f9815a05640 1 librados: starting msgr at 2024-03-25T18:22:06.055 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.052+0000 7f9815a05640 1 librados: starting objecter 2024-03-25T18:22:06.055 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.052+0000 7f9815a05640 1 Processor -- start 2024-03-25T18:22:06.055 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.052+0000 7f9815a05640 1 -- start start 2024-03-25T18:22:06.055 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.052+0000 7f9815a05640 1 librados: setting wanted keys 2024-03-25T18:22:06.055 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.052+0000 7f9815a05640 1 librados: calling monclient init 2024-03-25T18:22:06.055 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.052+0000 7f9815a05640 10 monclient: init 2024-03-25T18:22:06.056 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.053+0000 7f9815a05640 10 monclient: _reopen_session rank -1 2024-03-25T18:22:06.056 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.053+0000 7f9815a05640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:22:06.056 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.053+0000 7f9815a05640 10 monclient(hunting): picked mon.noname-b con 0x7f98100b4540 addr v1:172.21.15.98:6789/0 2024-03-25T18:22:06.057 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.053+0000 7f9815a05640 10 monclient(hunting): picked mon.noname-c con 0x7f98100b4f40 addr v1:172.21.15.97:6790/0 2024-03-25T18:22:06.057 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.053+0000 7f9815a05640 10 monclient(hunting): picked mon.noname-a con 0x7f981013fd90 addr v1:172.21.15.97:6789/0 2024-03-25T18:22:06.057 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.053+0000 7f9815a05640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98101db170 con 0x7f981013fd90 2024-03-25T18:22:06.057 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.053+0000 7f9815a05640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98100b0a30 con 0x7f98100b4540 2024-03-25T18:22:06.057 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.053+0000 7f9815a05640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98101dbd10 con 0x7f98100b4f40 2024-03-25T18:22:06.057 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.053+0000 7f9815a05640 10 monclient(hunting): _renew_subs 2024-03-25T18:22:06.057 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.053+0000 7f9815a05640 10 monclient(hunting): authenticate will time out at 7938.329328s 2024-03-25T18:22:06.060 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.059+0000 7f980effd640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f98100b4540 0x7f98100b1150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.97:34044/0 (socket says 172.21.15.97:34044) 2024-03-25T18:22:06.060 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.059+0000 7f980effd640 1 -- 172.21.15.97:0/2927784940 learned_addr learned my addr 172.21.15.97:0/2927784940 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:22:06.061 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.059+0000 7f97f7fff640 10 client.?.objecter ms_handle_connect 0x7f98100b4540 2024-03-25T18:22:06.061 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.059+0000 7f97f7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:06.063 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.062+0000 7f97f7fff640 10 client.?.objecter ms_handle_connect 0x7f98100b4f40 2024-03-25T18:22:06.064 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.062+0000 7f97f7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:06.066 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.065+0000 7f97f7fff640 10 client.?.objecter ms_handle_connect 0x7f981013fd90 2024-03-25T18:22:06.066 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.065+0000 7f97f7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:06.069 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.068+0000 7f97f7fff640 1 -- 172.21.15.97:0/2927784940 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 965257314 0 0) 0x7f98100b0a30 con 0x7f98100b4540 2024-03-25T18:22:06.069 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.068+0000 7f97f7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:22:06.069 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.068+0000 7f97f7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:06.069 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.068+0000 7f97f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:06.069 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.068+0000 7f97f7fff640 10 monclient(hunting): my global_id is 4486 2024-03-25T18:22:06.069 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.068+0000 7f97f7fff640 1 -- 172.21.15.97:0/2927784940 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97fc004eb0 con 0x7f98100b4540 2024-03-25T18:22:06.070 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.068+0000 7f97f7fff640 1 -- 172.21.15.97:0/2927784940 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3340446211 0 0) 0x7f98101db170 con 0x7f981013fd90 2024-03-25T18:22:06.070 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.068+0000 7f97f7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:22:06.070 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.068+0000 7f97f7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:06.070 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.068+0000 7f97f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:06.070 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.068+0000 7f97f7fff640 10 monclient(hunting): my global_id is 4482 2024-03-25T18:22:06.070 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.068+0000 7f97f7fff640 1 -- 172.21.15.97:0/2927784940 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98100b0a30 con 0x7f981013fd90 2024-03-25T18:22:06.076 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.074+0000 7f97f7fff640 1 -- 172.21.15.97:0/2927784940 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1395533653 0 0) 0x7f98101dbd10 con 0x7f98100b4f40 2024-03-25T18:22:06.076 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.075+0000 7f97f7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:22:06.076 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.075+0000 7f97f7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:06.076 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.075+0000 7f97f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:06.076 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.075+0000 7f97f7fff640 10 monclient(hunting): my global_id is 4478 2024-03-25T18:22:06.076 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.075+0000 7f97f7fff640 1 -- 172.21.15.97:0/2927784940 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98101db170 con 0x7f98100b4f40 2024-03-25T18:22:06.081 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.079+0000 7f97f7fff640 1 -- 172.21.15.97:0/2927784940 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2930400671 0 0) 0x7f97fc004eb0 con 0x7f98100b4540 2024-03-25T18:22:06.081 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.079+0000 7f97f7fff640 1 -- 172.21.15.97:0/2927784940 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f98101dbd10 con 0x7f98100b4540 2024-03-25T18:22:06.081 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.079+0000 7f97f7fff640 1 -- 172.21.15.97:0/2927784940 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f98000019f0 con 0x7f98100b4540 2024-03-25T18:22:06.081 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.079+0000 7f97f7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:06.081 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.079+0000 7f97f7fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:22:06.081 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.079+0000 7f97f7fff640 10 monclient(hunting): dump: 2024-03-25T18:22:06.081 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:22:06.081 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:22:06.081 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:22:06.081 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:22:06.081 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:06.081 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:22:06.081 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:22:06.082 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:22:06.082 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:22:06.082 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:22:06.082 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.079+0000 7f97f7fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:22:06.087 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.086+0000 7f97f7fff640 1 -- 172.21.15.97:0/2927784940 <== mon.2 v1:172.21.15.97:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2703492568 0 0) 0x7f98101db170 con 0x7f98100b4f40 2024-03-25T18:22:06.087 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.086+0000 7f97f7fff640 1 -- 172.21.15.97:0/2927784940 --> v1:172.21.15.97:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f97fc004eb0 con 0x7f98100b4f40 2024-03-25T18:22:06.087 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.086+0000 7f97f7fff640 1 -- 172.21.15.97:0/2927784940 <== mon.2 v1:172.21.15.97:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f9804001b00 con 0x7f98100b4f40 2024-03-25T18:22:06.087 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.086+0000 7f97f7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:06.087 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.086+0000 7f97f7fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:22:06.087 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.086+0000 7f97f7fff640 10 monclient(hunting): dump: 2024-03-25T18:22:06.087 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:22:06.087 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:22:06.088 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:22:06.088 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:22:06.088 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:06.088 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:22:06.088 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:22:06.088 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:22:06.088 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:22:06.088 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:22:06.094 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.091+0000 7f97f7fff640 1 -- 172.21.15.97:0/2927784940 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 722458483 0 0) 0x7f98100b0a30 con 0x7f981013fd90 2024-03-25T18:22:06.094 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.091+0000 7f97f7fff640 1 -- 172.21.15.97:0/2927784940 --> v1:172.21.15.97:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f98101db170 con 0x7f981013fd90 2024-03-25T18:22:06.094 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.091+0000 7f97f7fff640 1 -- 172.21.15.97:0/2927784940 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f9808001bf0 con 0x7f981013fd90 2024-03-25T18:22:06.094 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.091+0000 7f97f7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:06.094 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.091+0000 7f97f7fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:22:06.094 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.091+0000 7f97f7fff640 10 monclient(hunting): dump: 2024-03-25T18:22:06.094 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:22:06.094 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:22:06.094 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:22:06.094 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:22:06.095 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:06.095 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:22:06.095 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:22:06.095 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:22:06.095 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:22:06.095 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:22:06.095 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.091+0000 7f97f7fff640 1 -- 172.21.15.97:0/2927784940 <== mon.2 v1:172.21.15.97:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2456403354 0 0) 0x7f97fc004eb0 con 0x7f98100b4f40 2024-03-25T18:22:06.095 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.091+0000 7f97f7fff640 1 -- 172.21.15.97:0/2927784940 >> v1:172.21.15.98:6789/0 conn(0x7f98100b4540 legacy=0x7f98100b1150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:06.095 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.091+0000 7f97f7fff640 1 -- 172.21.15.97:0/2927784940 >> v1:172.21.15.97:6789/0 conn(0x7f981013fd90 legacy=0x7f9810137590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:06.095 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.091+0000 7f97f7fff640 10 monclient: _finish_hunting 0 2024-03-25T18:22:06.095 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.091+0000 7f97f7fff640 1 monclient: found mon.c 2024-03-25T18:22:06.095 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.091+0000 7f97f7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:22:06.095 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.091+0000 7f97f7fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.97:6790/0 2024-03-25T18:22:06.095 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.091+0000 7f97f7fff640 1 -- 172.21.15.97:0/2927784940 --> v1:172.21.15.97:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98101dadb0 con 0x7f98100b4f40 2024-03-25T18:22:06.095 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.091+0000 7f97f7fff640 10 monclient: _finish_auth 0 2024-03-25T18:22:06.096 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.091+0000 7f97f7fff640 10 monclient: _check_auth_tickets 2024-03-25T18:22:06.096 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.091+0000 7f97f7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:22:06.096 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.091+0000 7f9815a05640 5 monclient: authenticate success, global_id 4478 2024-03-25T18:22:06.096 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.091+0000 7f9815a05640 10 monclient: _renew_subs 2024-03-25T18:22:06.096 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.091+0000 7f9815a05640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.97:6790/0 2024-03-25T18:22:06.096 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.091+0000 7f9815a05640 1 -- 172.21.15.97:0/2927784940 --> v1:172.21.15.97:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9810137ca0 con 0x7f98100b4f40 2024-03-25T18:22:06.096 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.092+0000 7f9815a05640 10 client.4478.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:22:06.096 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.092+0000 7f9815a05640 10 monclient: _renew_subs 2024-03-25T18:22:06.096 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.092+0000 7f9815a05640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.97:6790/0 2024-03-25T18:22:06.096 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.092+0000 7f9815a05640 1 -- 172.21.15.97:0/2927784940 --> v1:172.21.15.97:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9810138350 con 0x7f98100b4f40 2024-03-25T18:22:06.096 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.092+0000 7f9815a05640 1 librados: init done 2024-03-25T18:22:06.096 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.092+0000 7f9815a05640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:06.096 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.092+0000 7f9815a05640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:06.096 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.092+0000 7f9815a05640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.97:6790/0 2024-03-25T18:22:06.096 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.092+0000 7f9815a05640 1 -- 172.21.15.97:0/2927784940 --> v1:172.21.15.97:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98100627b0 con 0x7f98100b4f40 2024-03-25T18:22:06.405 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.404+0000 7f97f7fff640 1 -- 172.21.15.97:0/2927784940 <== mon.2 v1:172.21.15.97:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9804001c80 con 0x7f98100b4f40 2024-03-25T18:22:06.405 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.404+0000 7f97f7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:22:06.406 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.404+0000 7f97f7fff640 1 -- 172.21.15.97:0/2927784940 <== mon.2 v1:172.21.15.97:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f9804003a40 con 0x7f98100b4f40 2024-03-25T18:22:06.406 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.404+0000 7f97f7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:06.406 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.404+0000 7f97f7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:22:06.406 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.404+0000 7f97f7fff640 10 monclient: dump: 2024-03-25T18:22:06.406 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:22:06.406 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:22:06.406 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:22:06.406 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:22:06.406 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:06.406 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:22:06.406 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:22:06.406 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:22:06.406 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:22:06.406 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:22:06.407 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.404+0000 7f980d7fa640 4 set_mon_vals no callback set 2024-03-25T18:22:06.407 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.404+0000 7f97f7fff640 1 -- 172.21.15.97:0/2927784940 <== mon.2 v1:172.21.15.97:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2377622137 0 0) 0x7f9804019a10 con 0x7f98100b4f40 2024-03-25T18:22:06.407 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.405+0000 7f980effd640 10 monclient: get_auth_request con 0x7f97fc06e110 auth_method 0 2024-03-25T18:22:06.407 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.405+0000 7f97f7fff640 10 client.4478.objecter ms_handle_connect 0x7f97fc06e110 2024-03-25T18:22:06.412 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.411+0000 7f97f7fff640 1 -- 172.21.15.97:0/2927784940 <== mon.2 v1:172.21.15.97:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 430711918 0 0) 0x7f9804053520 con 0x7f98100b4f40 2024-03-25T18:22:06.412 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.411+0000 7f97f7fff640 10 client.4478.objecter ms_dispatch 0x7f9810000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:22:06.412 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.411+0000 7f97f7fff640 3 client.4478.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:22:06.413 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.411+0000 7f97f7fff640 3 client.4478.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:22:06.413 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.411+0000 7f97f7fff640 20 client.4478.objecter dump_active .. 0 homeless 2024-03-25T18:22:06.413 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.411+0000 7f97f7fff640 1 -- 172.21.15.97:0/2927784940 <== mon.2 v1:172.21.15.97:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f980404f030 con 0x7f98100b4f40 2024-03-25T18:22:06.413 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.411+0000 7f97f7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:06.413 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.411+0000 7f97f7fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:22:06.673 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.671+0000 7f9815a05640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:06.673 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.671+0000 7f9815a05640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:06.673 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.671+0000 7f9815a05640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.97:6790/0 2024-03-25T18:22:06.673 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.671+0000 7f9815a05640 1 -- 172.21.15.97:0/2927784940 --> v1:172.21.15.97:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f98101db300 con 0x7f98100b4f40 2024-03-25T18:22:06.693 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.692+0000 7f97f7fff640 1 -- 172.21.15.97:0/2927784940 <== mon.2 v1:172.21.15.97:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2169427412) 0x7f9804050f60 con 0x7f98100b4f40 2024-03-25T18:22:06.693 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.692+0000 7f97f7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:06.693 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.692+0000 7f97f7fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:22:06.693 INFO:teuthology.misc.health.smithi097.stdout: 2024-03-25T18:22:06.693 INFO:teuthology.misc.health.smithi097.stdout:{"epoch":10,"fsid":"4c3d472c-79b8-486b-9e3d-2738d4306754","created":"2024-03-25T18:16:14.737309+0000","modified":"2024-03-25T18:16:20.357492+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:20.357492+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cb5152b9-262f-45c3-94b8-18d0e81df558","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6daf8793-0838-473d-a7d6-9a0d2894c75d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"136f1493-3b9c-4f43-a2e7-641482d69dae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"397d538c-4f64-4daf-9e8b-662d09f91327","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4f1df08-663f-4bda-93f8-3878ab7c24b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c4b18d70-a3ce-4b15-9b70-eae5525bd761","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8dd4033a-de0e-4f8a-96dc-59f3c1f825b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0077722e-ffa6-4a3d-9891-929d9869b286","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:22:06.696 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.694+0000 7f9815a05640 10 librados: watch_flush enter 2024-03-25T18:22:06.696 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.694+0000 7f9815a05640 10 librados: watch_flush exit 2024-03-25T18:22:06.696 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.694+0000 7f9815a05640 20 client.4478.objecter shutdown clearing up homeless session... 2024-03-25T18:22:06.696 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.694+0000 7f9815a05640 10 client.4478.objecter successfully canceled tick 2024-03-25T18:22:06.696 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.694+0000 7f9815a05640 1 -- 172.21.15.97:0/2927784940 >> v1:172.21.15.98:6800/800387573 conn(0x7f97fc06e110 legacy=0x7f97fc0705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:06.696 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.694+0000 7f9815a05640 10 monclient: shutdown 2024-03-25T18:22:06.696 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.694+0000 7f9815a05640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:22:06.696 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.694+0000 7f9815a05640 1 -- 172.21.15.97:0/2927784940 >> v1:172.21.15.97:6790/0 conn(0x7f98100b4f40 legacy=0x7f98101db600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:06.696 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.694+0000 7f9815a05640 1 -- 172.21.15.97:0/2927784940 shutdown_connections 2024-03-25T18:22:06.696 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.694+0000 7f9815a05640 1 -- 172.21.15.97:0/2927784940 >> 172.21.15.97:0/2927784940 conn(0x7f98100617a0 msgr2=0x7f9810139d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:22:06.697 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.695+0000 7f9815a05640 1 -- 172.21.15.97:0/2927784940 shutdown_connections 2024-03-25T18:22:06.697 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.695+0000 7f9815a05640 1 -- 172.21.15.97:0/2927784940 wait complete. 2024-03-25T18:22:06.697 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:06.695+0000 7f9815a05640 1 librados: shutdown 2024-03-25T18:22:06.708 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:22:12.967 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:06:15.006 34931== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:12.974 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:06:15.013 34931== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:13.006 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:06:15.045 34931== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:13.027 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:06:15.066 34935== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:13.027 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:06:15.070 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:13.031 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:06:15.070 34935== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:13.032 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:06:15.075 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:13.062 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:06:15.105 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:13.065 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:06:15.104 34935== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:13.512 DEBUG:teuthology.orchestra.run.smithi097:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:22:13.628 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:13.627+0000 7f1a92dcf640 10 monclient: get_monmap_and_config 2024-03-25T18:22:13.628 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:13.627+0000 7f1a92dcf640 10 monclient: build_initial_monmap 2024-03-25T18:22:13.628 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:13.627+0000 7f1a92dcf640 10 monclient: monmap: 2024-03-25T18:22:13.628 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:22:13.628 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:22:13.628 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:22:13.628326+0000 2024-03-25T18:22:13.628 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:22:13.628326+0000 2024-03-25T18:22:13.629 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:22:13.629 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:22:13.629 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:22:13.629 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T18:22:13.629 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-b 2024-03-25T18:22:13.629 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:22:13.630 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:13.628+0000 7f1a92dcf640 1 Processor -- start 2024-03-25T18:22:13.630 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:13.628+0000 7f1a92dcf640 1 -- start start 2024-03-25T18:22:13.630 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:13.628+0000 7f1a92dcf640 10 monclient: init 2024-03-25T18:22:13.630 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:13.628+0000 7f1a92dcf640 10 monclient: _reopen_session rank -1 2024-03-25T18:22:13.630 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:13.628+0000 7f1a92dcf640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:22:13.630 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:13.628+0000 7f1a92dcf640 10 monclient(hunting): picked mon.noname-a con 0x7f1a8c0a98f0 addr v1:172.21.15.97:6789/0 2024-03-25T18:22:13.630 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:13.628+0000 7f1a92dcf640 10 monclient(hunting): picked mon.noname-b con 0x7f1a8c13f710 addr v1:172.21.15.97:6790/0 2024-03-25T18:22:13.630 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:13.628+0000 7f1a92dcf640 10 monclient(hunting): picked mon.noname-c con 0x7f1a8c1430c0 addr v1:172.21.15.98:6789/0 2024-03-25T18:22:13.630 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:13.628+0000 7f1a92dcf640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a8c142520 con 0x7f1a8c0a98f0 2024-03-25T18:22:13.630 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:13.628+0000 7f1a92dcf640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a8c0b28a0 con 0x7f1a8c1430c0 2024-03-25T18:22:13.631 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:13.628+0000 7f1a92dcf640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a8c0b2a30 con 0x7f1a8c13f710 2024-03-25T18:22:13.631 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:13.628+0000 7f1a92dcf640 10 monclient(hunting): _renew_subs 2024-03-25T18:22:13.631 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:13.628+0000 7f1a92dcf640 10 monclient(hunting): authenticate will time out at 7945.904696s 2024-03-25T18:22:13.639 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:13.638+0000 7f1a90b6b640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f1a8c0a98f0 0x7f1a8c0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:39868/0 (socket says 172.21.15.97:39868) 2024-03-25T18:22:13.639 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:13.638+0000 7f1a90b6b640 1 -- 172.21.15.97:0/1495493008 learned_addr learned my addr 172.21.15.97:0/1495493008 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:22:13.645 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:13.644+0000 7f1a8b7fe640 1 -- 172.21.15.97:0/1495493008 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 256151015 0 0) 0x7f1a8c142520 con 0x7f1a8c0a98f0 2024-03-25T18:22:13.646 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:13.644+0000 7f1a8b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:22:13.646 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:13.644+0000 7f1a8b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:13.646 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:13.644+0000 7f1a8b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:13.646 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:13.644+0000 7f1a8b7fe640 10 monclient(hunting): my global_id is 4485 2024-03-25T18:22:13.646 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:13.644+0000 7f1a8b7fe640 1 -- 172.21.15.97:0/1495493008 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a7c003520 con 0x7f1a8c0a98f0 2024-03-25T18:22:13.658 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:13.656+0000 7f1a8b7fe640 1 -- 172.21.15.97:0/1495493008 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3234741363 0 0) 0x7f1a8c0b2a30 con 0x7f1a8c13f710 2024-03-25T18:22:13.658 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:13.656+0000 7f1a8b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:22:13.658 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:13.656+0000 7f1a8b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:13.658 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:13.656+0000 7f1a8b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:13.658 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:13.656+0000 7f1a8b7fe640 10 monclient(hunting): my global_id is 4481 2024-03-25T18:22:13.658 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:13.656+0000 7f1a8b7fe640 1 -- 172.21.15.97:0/1495493008 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a8c142520 con 0x7f1a8c13f710 2024-03-25T18:22:13.661 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:13.660+0000 7f1a8b7fe640 1 -- 172.21.15.97:0/1495493008 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1726973247 0 0) 0x7f1a7c003520 con 0x7f1a8c0a98f0 2024-03-25T18:22:13.662 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:13.660+0000 7f1a8b7fe640 1 -- 172.21.15.97:0/1495493008 >> v1:172.21.15.97:6790/0 conn(0x7f1a8c13f710 legacy=0x7f1a8c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:13.662 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:13.660+0000 7f1a8b7fe640 1 -- 172.21.15.97:0/1495493008 >> v1:172.21.15.98:6789/0 conn(0x7f1a8c1430c0 legacy=0x7f1a8c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:13.662 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:13.660+0000 7f1a8b7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:22:13.662 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:13.660+0000 7f1a8b7fe640 1 monclient: found mon.noname-a 2024-03-25T18:22:13.662 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:13.660+0000 7f1a8b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:22:13.662 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:13.660+0000 7f1a8b7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.97:6789/0 2024-03-25T18:22:13.662 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:13.660+0000 7f1a8b7fe640 1 -- 172.21.15.97:0/1495493008 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a8c0b44c0 con 0x7f1a8c0a98f0 2024-03-25T18:22:13.662 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:13.660+0000 7f1a8b7fe640 10 monclient: _finish_auth 0 2024-03-25T18:22:13.662 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:13.660+0000 7f1a8b7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:22:13.662 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:13.660+0000 7f1a8b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:22:13.662 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:13.660+0000 7f1a8b7fe640 1 -- 172.21.15.97:0/1495493008 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f1a80001690 con 0x7f1a8c0a98f0 2024-03-25T18:22:13.662 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:13.660+0000 7f1a8b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:13.662 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:13.660+0000 7f1a8b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:22:13.663 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:13.660+0000 7f1a8b7fe640 10 monclient: dump: 2024-03-25T18:22:13.663 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:22:13.663 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:22:13.663 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:22:13.663 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:22:13.663 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:13.663 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:22:13.663 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:22:13.663 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:22:13.663 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:22:13.663 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:22:13.663 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:13.660+0000 7f1a92dcf640 5 monclient: authenticate success, global_id 4485 2024-03-25T18:22:13.663 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:13.660+0000 7f1a92dcf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:22:13.671 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:13.670+0000 7f1a8b7fe640 1 -- 172.21.15.97:0/1495493008 <== mon.0 v1:172.21.15.97:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1a800017f0 con 0x7f1a8c0a98f0 2024-03-25T18:22:13.672 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:13.670+0000 7f1a8b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:22:13.672 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:13.670+0000 7f1a8b7fe640 1 -- 172.21.15.97:0/1495493008 <== mon.0 v1:172.21.15.97:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f1a80001c00 con 0x7f1a8c0a98f0 2024-03-25T18:22:13.672 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:13.670+0000 7f1a8b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:13.672 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:13.670+0000 7f1a8b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:22:13.672 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:13.670+0000 7f1a8b7fe640 10 monclient: dump: 2024-03-25T18:22:13.672 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:22:13.672 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:22:13.672 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:22:13.672 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:22:13.672 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:13.672 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:22:13.673 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:22:13.673 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:22:13.673 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:22:13.673 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:22:13.673 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:13.670+0000 7f1a92dcf640 10 monclient: get_monmap_and_config success 2024-03-25T18:22:13.673 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:13.670+0000 7f1a92dcf640 4 set_mon_vals no callback set 2024-03-25T18:22:13.674 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:13.671+0000 7f1a92dcf640 10 monclient: shutdown 2024-03-25T18:22:13.674 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:13.671+0000 7f1a92dcf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:22:13.674 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:13.671+0000 7f1a92dcf640 1 -- 172.21.15.97:0/1495493008 >> v1:172.21.15.97:6789/0 conn(0x7f1a8c0a98f0 legacy=0x7f1a8c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:13.675 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:13.671+0000 7f1a92dcf640 1 -- 172.21.15.97:0/1495493008 shutdown_connections 2024-03-25T18:22:13.675 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:13.671+0000 7f1a92dcf640 1 -- 172.21.15.97:0/1495493008 >> 172.21.15.97:0/1495493008 conn(0x7f1a8c0b7ea0 msgr2=0x7f1a8c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:22:13.675 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:13.671+0000 7f1a92dcf640 1 -- 172.21.15.97:0/1495493008 shutdown_connections 2024-03-25T18:22:13.675 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:13.671+0000 7f1a92dcf640 1 -- 172.21.15.97:0/1495493008 wait complete. 2024-03-25T18:22:13.675 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:13.672+0000 7f1a92dcf640 10 monclient: build_initial_monmap 2024-03-25T18:22:13.675 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:13.672+0000 7f1a92dcf640 10 monclient: monmap: 2024-03-25T18:22:13.675 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:22:13.675 INFO:teuthology.misc.health.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:22:13.675 INFO:teuthology.misc.health.smithi097.stderr:last_changed 0.000000 2024-03-25T18:22:13.675 INFO:teuthology.misc.health.smithi097.stderr:created 0.000000 2024-03-25T18:22:13.675 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:22:13.675 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 1 2024-03-25T18:22:13.675 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:22:13.676 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T18:22:13.676 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-c 2024-03-25T18:22:13.676 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:22:13.676 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:13.672+0000 7f1a92dcf640 1 librados: starting msgr at 2024-03-25T18:22:13.676 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:13.672+0000 7f1a92dcf640 1 librados: starting objecter 2024-03-25T18:22:13.676 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:13.673+0000 7f1a92dcf640 1 Processor -- start 2024-03-25T18:22:13.676 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:13.673+0000 7f1a92dcf640 1 -- start start 2024-03-25T18:22:13.676 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:13.673+0000 7f1a92dcf640 1 librados: setting wanted keys 2024-03-25T18:22:13.676 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:13.673+0000 7f1a92dcf640 1 librados: calling monclient init 2024-03-25T18:22:13.676 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:13.673+0000 7f1a92dcf640 10 monclient: init 2024-03-25T18:22:13.677 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:13.673+0000 7f1a92dcf640 10 monclient: _reopen_session rank -1 2024-03-25T18:22:13.677 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:13.673+0000 7f1a92dcf640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:22:13.677 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:13.673+0000 7f1a92dcf640 10 monclient(hunting): picked mon.noname-c con 0x7f1a8c0a98f0 addr v1:172.21.15.97:6790/0 2024-03-25T18:22:13.678 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:13.673+0000 7f1a92dcf640 10 monclient(hunting): picked mon.noname-a con 0x7f1a8c13f710 addr v1:172.21.15.97:6789/0 2024-03-25T18:22:13.678 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:13.673+0000 7f1a92dcf640 10 monclient(hunting): picked mon.noname-b con 0x7f1a8c1430c0 addr v1:172.21.15.98:6789/0 2024-03-25T18:22:13.678 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:13.673+0000 7f1a92dcf640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a8c14d590 con 0x7f1a8c13f710 2024-03-25T18:22:13.678 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:13.673+0000 7f1a92dcf640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a8c0b78e0 con 0x7f1a8c1430c0 2024-03-25T18:22:13.678 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:13.673+0000 7f1a92dcf640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a8c0b7550 con 0x7f1a8c0a98f0 2024-03-25T18:22:13.678 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:13.673+0000 7f1a92dcf640 10 monclient(hunting): _renew_subs 2024-03-25T18:22:13.678 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:13.673+0000 7f1a92dcf640 10 monclient(hunting): authenticate will time out at 7945.949825s 2024-03-25T18:22:13.683 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:13.681+0000 7f1a9136c640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f1a8c1430c0 0x7f1a8c1db790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.97:52638/0 (socket says 172.21.15.97:52638) 2024-03-25T18:22:13.683 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:13.681+0000 7f1a9136c640 1 -- 172.21.15.97:0/3395033077 learned_addr learned my addr 172.21.15.97:0/3395033077 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:22:13.684 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:13.682+0000 7f1a897fa640 10 client.?.objecter ms_handle_connect 0x7f1a8c1430c0 2024-03-25T18:22:13.684 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:13.682+0000 7f1a897fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:13.690 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:13.689+0000 7f1a897fa640 10 client.?.objecter ms_handle_connect 0x7f1a8c0a98f0 2024-03-25T18:22:13.690 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:13.689+0000 7f1a897fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:13.695 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:13.694+0000 7f1a897fa640 1 -- 172.21.15.97:0/3395033077 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3702345326 0 0) 0x7f1a8c0b7550 con 0x7f1a8c0a98f0 2024-03-25T18:22:13.695 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:13.694+0000 7f1a897fa640 10 monclient(hunting): _negotiate 2024-03-25T18:22:13.696 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:13.694+0000 7f1a897fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:13.696 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:13.694+0000 7f1a897fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:13.696 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:13.694+0000 7f1a897fa640 10 monclient(hunting): my global_id is 4484 2024-03-25T18:22:13.696 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:13.694+0000 7f1a897fa640 1 -- 172.21.15.97:0/3395033077 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a70004eb0 con 0x7f1a8c0a98f0 2024-03-25T18:22:13.698 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:13.696+0000 7f1a897fa640 10 client.?.objecter ms_handle_connect 0x7f1a8c13f710 2024-03-25T18:22:13.698 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:13.696+0000 7f1a897fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:13.698 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:13.697+0000 7f1a897fa640 1 -- 172.21.15.97:0/3395033077 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3263136067 0 0) 0x7f1a8c0b78e0 con 0x7f1a8c1430c0 2024-03-25T18:22:13.699 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:13.697+0000 7f1a897fa640 10 monclient(hunting): _negotiate 2024-03-25T18:22:13.699 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:13.697+0000 7f1a897fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:13.699 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:13.697+0000 7f1a897fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:13.699 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:13.697+0000 7f1a897fa640 10 monclient(hunting): my global_id is 4492 2024-03-25T18:22:13.699 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:13.697+0000 7f1a897fa640 1 -- 172.21.15.97:0/3395033077 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a8c0b7550 con 0x7f1a8c1430c0 2024-03-25T18:22:13.705 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:13.704+0000 7f1a897fa640 1 -- 172.21.15.97:0/3395033077 <== mon.2 v1:172.21.15.97:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 707277633 0 0) 0x7f1a70004eb0 con 0x7f1a8c0a98f0 2024-03-25T18:22:13.706 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:13.704+0000 7f1a897fa640 1 -- 172.21.15.97:0/3395033077 --> v1:172.21.15.97:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1a8c0b78e0 con 0x7f1a8c0a98f0 2024-03-25T18:22:13.706 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:13.704+0000 7f1a897fa640 1 -- 172.21.15.97:0/3395033077 <== mon.2 v1:172.21.15.97:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f1a80001b00 con 0x7f1a8c0a98f0 2024-03-25T18:22:13.706 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:13.704+0000 7f1a897fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:13.706 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:13.704+0000 7f1a897fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:22:13.706 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:13.704+0000 7f1a897fa640 10 monclient(hunting): dump: 2024-03-25T18:22:13.706 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:22:13.706 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:22:13.706 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:22:13.706 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:22:13.706 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:13.706 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:22:13.706 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:22:13.706 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:22:13.707 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:22:13.707 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:22:13.707 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:13.704+0000 7f1a897fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:22:13.708 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:13.707+0000 7f1a897fa640 1 -- 172.21.15.97:0/3395033077 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3158483004 0 0) 0x7f1a8c14d590 con 0x7f1a8c13f710 2024-03-25T18:22:13.708 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:13.707+0000 7f1a897fa640 10 monclient(hunting): _negotiate 2024-03-25T18:22:13.708 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:13.707+0000 7f1a897fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:13.708 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:13.707+0000 7f1a897fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:13.708 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:13.707+0000 7f1a897fa640 10 monclient(hunting): my global_id is 4488 2024-03-25T18:22:13.709 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:13.707+0000 7f1a897fa640 1 -- 172.21.15.97:0/3395033077 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a70004eb0 con 0x7f1a8c13f710 2024-03-25T18:22:13.709 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:13.708+0000 7f1a897fa640 1 -- 172.21.15.97:0/3395033077 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2216184151 0 0) 0x7f1a8c0b7550 con 0x7f1a8c1430c0 2024-03-25T18:22:13.709 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:13.708+0000 7f1a897fa640 1 -- 172.21.15.97:0/3395033077 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1a8c14d590 con 0x7f1a8c1430c0 2024-03-25T18:22:13.709 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:13.708+0000 7f1a897fa640 1 -- 172.21.15.97:0/3395033077 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f1a84001bd0 con 0x7f1a8c1430c0 2024-03-25T18:22:13.709 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:13.708+0000 7f1a897fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:13.709 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:13.708+0000 7f1a897fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:22:13.710 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:13.708+0000 7f1a897fa640 10 monclient(hunting): dump: 2024-03-25T18:22:13.710 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:22:13.710 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:22:13.710 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:22:13.710 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:22:13.710 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:13.710 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:22:13.710 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:22:13.710 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:22:13.710 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:22:13.710 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:22:13.713 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:13.712+0000 7f1a897fa640 1 -- 172.21.15.97:0/3395033077 <== mon.2 v1:172.21.15.97:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 570660541 0 0) 0x7f1a8c0b78e0 con 0x7f1a8c0a98f0 2024-03-25T18:22:13.714 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:13.712+0000 7f1a897fa640 1 -- 172.21.15.97:0/3395033077 >> v1:172.21.15.98:6789/0 conn(0x7f1a8c1430c0 legacy=0x7f1a8c1db790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:13.714 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:13.712+0000 7f1a897fa640 1 -- 172.21.15.97:0/3395033077 >> v1:172.21.15.97:6789/0 conn(0x7f1a8c13f710 legacy=0x7f1a8c135120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:13.714 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:13.712+0000 7f1a897fa640 10 monclient: _finish_hunting 0 2024-03-25T18:22:13.714 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:13.712+0000 7f1a897fa640 1 monclient: found mon.c 2024-03-25T18:22:13.714 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:13.712+0000 7f1a897fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:22:13.714 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:13.712+0000 7f1a897fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.97:6790/0 2024-03-25T18:22:13.714 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:13.712+0000 7f1a897fa640 1 -- 172.21.15.97:0/3395033077 --> v1:172.21.15.97:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a8c135830 con 0x7f1a8c0a98f0 2024-03-25T18:22:13.714 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:13.712+0000 7f1a897fa640 10 monclient: _finish_auth 0 2024-03-25T18:22:13.714 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:13.712+0000 7f1a897fa640 10 monclient: _check_auth_tickets 2024-03-25T18:22:13.714 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:13.712+0000 7f1a897fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:22:13.714 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:13.712+0000 7f1a92dcf640 5 monclient: authenticate success, global_id 4484 2024-03-25T18:22:13.714 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:13.712+0000 7f1a92dcf640 10 monclient: _renew_subs 2024-03-25T18:22:13.714 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:13.712+0000 7f1a92dcf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.97:6790/0 2024-03-25T18:22:13.715 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:13.712+0000 7f1a92dcf640 1 -- 172.21.15.97:0/3395033077 --> v1:172.21.15.97:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a8c13afe0 con 0x7f1a8c0a98f0 2024-03-25T18:22:13.715 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:13.712+0000 7f1a92dcf640 10 client.4484.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:22:13.715 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:13.712+0000 7f1a92dcf640 10 monclient: _renew_subs 2024-03-25T18:22:13.715 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:13.712+0000 7f1a92dcf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.97:6790/0 2024-03-25T18:22:13.715 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:13.712+0000 7f1a92dcf640 1 -- 172.21.15.97:0/3395033077 --> v1:172.21.15.97:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a8c1dc080 con 0x7f1a8c0a98f0 2024-03-25T18:22:13.715 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:13.712+0000 7f1a92dcf640 1 librados: init done 2024-03-25T18:22:13.715 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:13.713+0000 7f1a92dcf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:13.716 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:13.713+0000 7f1a92dcf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:13.716 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:13.713+0000 7f1a92dcf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.97:6790/0 2024-03-25T18:22:13.716 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:13.713+0000 7f1a92dcf640 1 -- 172.21.15.97:0/3395033077 --> v1:172.21.15.97:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a8c135a40 con 0x7f1a8c0a98f0 2024-03-25T18:22:14.007 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:14.006+0000 7f1a897fa640 1 -- 172.21.15.97:0/3395033077 <== mon.2 v1:172.21.15.97:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1a80001220 con 0x7f1a8c0a98f0 2024-03-25T18:22:14.008 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:14.006+0000 7f1a897fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:22:14.008 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:14.006+0000 7f1a897fa640 1 -- 172.21.15.97:0/3395033077 <== mon.2 v1:172.21.15.97:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f1a80004a90 con 0x7f1a8c0a98f0 2024-03-25T18:22:14.008 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:14.006+0000 7f1a897fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:14.008 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:14.006+0000 7f1a897fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:22:14.008 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:14.006+0000 7f1a897fa640 10 monclient: dump: 2024-03-25T18:22:14.008 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:22:14.008 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:22:14.008 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:22:14.008 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:22:14.008 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:14.008 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:22:14.008 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:22:14.008 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:22:14.008 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:22:14.008 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:22:14.009 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:14.006+0000 7f1a8a7fc640 4 set_mon_vals no callback set 2024-03-25T18:22:14.009 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:14.006+0000 7f1a897fa640 1 -- 172.21.15.97:0/3395033077 <== mon.2 v1:172.21.15.97:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2377622137 0 0) 0x7f1a8001aa60 con 0x7f1a8c0a98f0 2024-03-25T18:22:14.009 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:14.007+0000 7f1a8bfff640 10 monclient: get_auth_request con 0x7f1a7006dfd0 auth_method 0 2024-03-25T18:22:14.009 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:14.007+0000 7f1a897fa640 10 client.4484.objecter ms_handle_connect 0x7f1a7006dfd0 2024-03-25T18:22:14.013 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:14.011+0000 7f1a897fa640 1 -- 172.21.15.97:0/3395033077 <== mon.2 v1:172.21.15.97:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 430711918 0 0) 0x7f1a80054650 con 0x7f1a8c0a98f0 2024-03-25T18:22:14.013 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:14.011+0000 7f1a897fa640 10 client.4484.objecter ms_dispatch 0x7f1a8c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:22:14.013 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:14.011+0000 7f1a897fa640 3 client.4484.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:22:14.013 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:14.011+0000 7f1a897fa640 3 client.4484.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:22:14.013 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:14.012+0000 7f1a897fa640 20 client.4484.objecter dump_active .. 0 homeless 2024-03-25T18:22:14.013 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:14.012+0000 7f1a897fa640 1 -- 172.21.15.97:0/3395033077 <== mon.2 v1:172.21.15.97:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1a8004ff90 con 0x7f1a8c0a98f0 2024-03-25T18:22:14.014 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:14.012+0000 7f1a897fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:14.014 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:14.012+0000 7f1a897fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:22:14.236 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:14.234+0000 7f1a92dcf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:14.236 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:14.234+0000 7f1a92dcf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:14.236 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:14.234+0000 7f1a92dcf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.97:6790/0 2024-03-25T18:22:14.236 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:14.234+0000 7f1a92dcf640 1 -- 172.21.15.97:0/3395033077 --> v1:172.21.15.97:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1a8c13b380 con 0x7f1a8c0a98f0 2024-03-25T18:22:14.265 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:14.263+0000 7f1a897fa640 1 -- 172.21.15.97:0/3395033077 <== mon.2 v1:172.21.15.97:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2169427412) 0x7f1a80052090 con 0x7f1a8c0a98f0 2024-03-25T18:22:14.265 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:14.263+0000 7f1a897fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:14.265 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:14.263+0000 7f1a897fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:22:14.265 INFO:teuthology.misc.health.smithi097.stdout: 2024-03-25T18:22:14.265 INFO:teuthology.misc.health.smithi097.stdout:{"epoch":10,"fsid":"4c3d472c-79b8-486b-9e3d-2738d4306754","created":"2024-03-25T18:16:14.737309+0000","modified":"2024-03-25T18:16:20.357492+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:20.357492+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cb5152b9-262f-45c3-94b8-18d0e81df558","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6daf8793-0838-473d-a7d6-9a0d2894c75d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"136f1493-3b9c-4f43-a2e7-641482d69dae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"397d538c-4f64-4daf-9e8b-662d09f91327","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4f1df08-663f-4bda-93f8-3878ab7c24b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c4b18d70-a3ce-4b15-9b70-eae5525bd761","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8dd4033a-de0e-4f8a-96dc-59f3c1f825b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0077722e-ffa6-4a3d-9891-929d9869b286","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:22:14.267 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:14.265+0000 7f1a92dcf640 10 librados: watch_flush enter 2024-03-25T18:22:14.267 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:14.265+0000 7f1a92dcf640 10 librados: watch_flush exit 2024-03-25T18:22:14.267 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:14.265+0000 7f1a92dcf640 20 client.4484.objecter shutdown clearing up homeless session... 2024-03-25T18:22:14.268 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:14.265+0000 7f1a92dcf640 10 client.4484.objecter successfully canceled tick 2024-03-25T18:22:14.268 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:14.266+0000 7f1a92dcf640 1 -- 172.21.15.97:0/3395033077 >> v1:172.21.15.98:6800/800387573 conn(0x7f1a7006dfd0 legacy=0x7f1a70070490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:14.268 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:14.266+0000 7f1a92dcf640 10 monclient: shutdown 2024-03-25T18:22:14.268 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:14.266+0000 7f1a92dcf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:22:14.268 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:14.266+0000 7f1a92dcf640 1 -- 172.21.15.97:0/3395033077 >> v1:172.21.15.97:6790/0 conn(0x7f1a8c0a98f0 legacy=0x7f1a8c14c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:14.268 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:14.266+0000 7f1a92dcf640 1 -- 172.21.15.97:0/3395033077 shutdown_connections 2024-03-25T18:22:14.268 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:14.266+0000 7f1a92dcf640 1 -- 172.21.15.97:0/3395033077 >> 172.21.15.97:0/3395033077 conn(0x7f1a8c0b7ea0 msgr2=0x7f1a8c13c640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:22:14.268 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:14.266+0000 7f1a92dcf640 1 -- 172.21.15.97:0/3395033077 shutdown_connections 2024-03-25T18:22:14.268 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:14.266+0000 7f1a92dcf640 1 -- 172.21.15.97:0/3395033077 wait complete. 2024-03-25T18:22:14.268 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:14.266+0000 7f1a92dcf640 1 librados: shutdown 2024-03-25T18:22:14.279 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:22:21.086 DEBUG:teuthology.orchestra.run.smithi097:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:22:21.200 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.200+0000 7f32800ff640 10 monclient: get_monmap_and_config 2024-03-25T18:22:21.200 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.200+0000 7f32800ff640 10 monclient: build_initial_monmap 2024-03-25T18:22:21.201 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.200+0000 7f32800ff640 10 monclient: monmap: 2024-03-25T18:22:21.201 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:22:21.201 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:22:21.201 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:22:21.200332+0000 2024-03-25T18:22:21.201 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:22:21.200332+0000 2024-03-25T18:22:21.201 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:22:21.201 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:22:21.201 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:22:21.201 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T18:22:21.201 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-b 2024-03-25T18:22:21.201 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:22:21.202 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.200+0000 7f32800ff640 1 Processor -- start 2024-03-25T18:22:21.202 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.201+0000 7f32800ff640 1 -- start start 2024-03-25T18:22:21.202 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.201+0000 7f32800ff640 10 monclient: init 2024-03-25T18:22:21.203 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.201+0000 7f32800ff640 10 monclient: _reopen_session rank -1 2024-03-25T18:22:21.203 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.201+0000 7f32800ff640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:22:21.203 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.201+0000 7f32800ff640 10 monclient(hunting): picked mon.noname-c con 0x7f32780a98f0 addr v1:172.21.15.98:6789/0 2024-03-25T18:22:21.203 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.201+0000 7f32800ff640 10 monclient(hunting): picked mon.noname-a con 0x7f327813f710 addr v1:172.21.15.97:6789/0 2024-03-25T18:22:21.203 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.201+0000 7f32800ff640 10 monclient(hunting): picked mon.noname-b con 0x7f32781430c0 addr v1:172.21.15.97:6790/0 2024-03-25T18:22:21.203 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.201+0000 7f32800ff640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3278142520 con 0x7f327813f710 2024-03-25T18:22:21.203 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.201+0000 7f32800ff640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32780b28a0 con 0x7f32780a98f0 2024-03-25T18:22:21.203 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.201+0000 7f32800ff640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32780b2a30 con 0x7f32781430c0 2024-03-25T18:22:21.203 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.201+0000 7f32800ff640 10 monclient(hunting): _renew_subs 2024-03-25T18:22:21.203 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.201+0000 7f32800ff640 10 monclient(hunting): authenticate will time out at 7953.476779s 2024-03-25T18:22:21.218 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.218+0000 7f327d69a640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f327813f710 0x7f32781429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:58456/0 (socket says 172.21.15.97:58456) 2024-03-25T18:22:21.218 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.218+0000 7f327d69a640 1 -- 172.21.15.97:0/1044771025 learned_addr learned my addr 172.21.15.97:0/1044771025 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:22:21.229 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.229+0000 7f327ce99640 1 -- 172.21.15.97:0/1044771025 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1441361630 0 0) 0x7f3278142520 con 0x7f327813f710 2024-03-25T18:22:21.229 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.229+0000 7f327ce99640 10 monclient(hunting): _negotiate 2024-03-25T18:22:21.230 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.229+0000 7f327ce99640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:21.230 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.229+0000 7f327ce99640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:21.230 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.229+0000 7f327ce99640 10 monclient(hunting): my global_id is 4491 2024-03-25T18:22:21.230 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.229+0000 7f327ce99640 1 -- 172.21.15.97:0/1044771025 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3264003520 con 0x7f327813f710 2024-03-25T18:22:21.236 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.236+0000 7f327ce99640 1 -- 172.21.15.97:0/1044771025 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3572856917 0 0) 0x7f32780b2a30 con 0x7f32781430c0 2024-03-25T18:22:21.236 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.236+0000 7f327ce99640 10 monclient(hunting): _negotiate 2024-03-25T18:22:21.236 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.236+0000 7f327ce99640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:21.236 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.236+0000 7f327ce99640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:21.236 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.236+0000 7f327ce99640 10 monclient(hunting): my global_id is 4487 2024-03-25T18:22:21.236 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.236+0000 7f327ce99640 1 -- 172.21.15.97:0/1044771025 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3278142520 con 0x7f32781430c0 2024-03-25T18:22:21.256 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.256+0000 7f327ce99640 1 -- 172.21.15.97:0/1044771025 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1876952352 0 0) 0x7f32780b28a0 con 0x7f32780a98f0 2024-03-25T18:22:21.256 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.256+0000 7f327ce99640 10 monclient(hunting): _negotiate 2024-03-25T18:22:21.256 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.256+0000 7f327ce99640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:21.257 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.256+0000 7f327ce99640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:21.257 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.256+0000 7f327ce99640 10 monclient(hunting): my global_id is 4495 2024-03-25T18:22:21.257 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.256+0000 7f327ce99640 1 -- 172.21.15.97:0/1044771025 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32780b2a30 con 0x7f32780a98f0 2024-03-25T18:22:21.259 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.258+0000 7f327ce99640 1 -- 172.21.15.97:0/1044771025 <== mon.2 v1:172.21.15.97:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 925227141 0 0) 0x7f3278142520 con 0x7f32781430c0 2024-03-25T18:22:21.259 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.258+0000 7f327ce99640 1 -- 172.21.15.97:0/1044771025 >> v1:172.21.15.98:6789/0 conn(0x7f32780a98f0 legacy=0x7f32780a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:21.259 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.258+0000 7f327ce99640 1 -- 172.21.15.97:0/1044771025 >> v1:172.21.15.97:6789/0 conn(0x7f327813f710 legacy=0x7f32781429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:21.259 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.258+0000 7f327ce99640 10 monclient: _finish_hunting 0 2024-03-25T18:22:21.259 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.258+0000 7f327ce99640 1 monclient: found mon.noname-b 2024-03-25T18:22:21.259 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.258+0000 7f327ce99640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:22:21.259 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.258+0000 7f327ce99640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.97:6790/0 2024-03-25T18:22:21.259 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.258+0000 7f327ce99640 1 -- 172.21.15.97:0/1044771025 --> v1:172.21.15.97:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32780b44c0 con 0x7f32781430c0 2024-03-25T18:22:21.259 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.258+0000 7f327ce99640 10 monclient: _finish_auth 0 2024-03-25T18:22:21.260 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.258+0000 7f327ce99640 10 monclient: _check_auth_tickets 2024-03-25T18:22:21.260 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.258+0000 7f327ce99640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:22:21.260 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.258+0000 7f327ce99640 1 -- 172.21.15.97:0/1044771025 <== mon.2 v1:172.21.15.97:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f32700019f0 con 0x7f32781430c0 2024-03-25T18:22:21.260 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.258+0000 7f327ce99640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:21.260 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.258+0000 7f327ce99640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:22:21.260 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.258+0000 7f327ce99640 10 monclient: dump: 2024-03-25T18:22:21.260 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:22:21.260 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:22:21.260 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:22:21.260 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:22:21.260 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:21.260 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:22:21.261 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:22:21.261 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:22:21.261 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:22:21.261 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:22:21.261 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.258+0000 7f32800ff640 5 monclient: authenticate success, global_id 4487 2024-03-25T18:22:21.261 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.258+0000 7f32800ff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:22:21.275 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.275+0000 7f327ce99640 1 -- 172.21.15.97:0/1044771025 <== mon.2 v1:172.21.15.97:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3270001b50 con 0x7f32781430c0 2024-03-25T18:22:21.276 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.275+0000 7f327ce99640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:22:21.276 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.275+0000 7f327ce99640 1 -- 172.21.15.97:0/1044771025 <== mon.2 v1:172.21.15.97:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f3270001f60 con 0x7f32781430c0 2024-03-25T18:22:21.276 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.275+0000 7f32800ff640 10 monclient: get_monmap_and_config success 2024-03-25T18:22:21.276 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.275+0000 7f32800ff640 4 set_mon_vals no callback set 2024-03-25T18:22:21.276 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.275+0000 7f327ce99640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:21.276 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.275+0000 7f327ce99640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:22:21.276 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.275+0000 7f327ce99640 10 monclient: dump: 2024-03-25T18:22:21.277 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:22:21.277 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:22:21.277 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:22:21.277 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:22:21.277 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:21.277 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:22:21.278 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:22:21.278 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:22:21.278 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:22:21.278 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:22:21.281 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.276+0000 7f32800ff640 10 monclient: shutdown 2024-03-25T18:22:21.281 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.276+0000 7f32800ff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:22:21.281 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.276+0000 7f32800ff640 1 -- 172.21.15.97:0/1044771025 >> v1:172.21.15.97:6790/0 conn(0x7f32781430c0 legacy=0x7f327814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:21.281 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.276+0000 7f32800ff640 1 -- 172.21.15.97:0/1044771025 shutdown_connections 2024-03-25T18:22:21.281 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.276+0000 7f32800ff640 1 -- 172.21.15.97:0/1044771025 >> 172.21.15.97:0/1044771025 conn(0x7f32780b7ea0 msgr2=0x7f32780b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:22:21.281 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.276+0000 7f32800ff640 1 -- 172.21.15.97:0/1044771025 shutdown_connections 2024-03-25T18:22:21.282 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.276+0000 7f32800ff640 1 -- 172.21.15.97:0/1044771025 wait complete. 2024-03-25T18:22:21.282 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.277+0000 7f32800ff640 10 monclient: build_initial_monmap 2024-03-25T18:22:21.282 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.277+0000 7f32800ff640 10 monclient: monmap: 2024-03-25T18:22:21.282 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:22:21.282 INFO:teuthology.misc.health.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:22:21.282 INFO:teuthology.misc.health.smithi097.stderr:last_changed 0.000000 2024-03-25T18:22:21.282 INFO:teuthology.misc.health.smithi097.stderr:created 0.000000 2024-03-25T18:22:21.282 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:22:21.282 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 1 2024-03-25T18:22:21.282 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:22:21.282 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T18:22:21.282 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-c 2024-03-25T18:22:21.282 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:22:21.283 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.277+0000 7f32800ff640 1 librados: starting msgr at 2024-03-25T18:22:21.283 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.277+0000 7f32800ff640 1 librados: starting objecter 2024-03-25T18:22:21.283 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.277+0000 7f32800ff640 1 Processor -- start 2024-03-25T18:22:21.283 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.277+0000 7f32800ff640 1 -- start start 2024-03-25T18:22:21.283 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.277+0000 7f32800ff640 1 librados: setting wanted keys 2024-03-25T18:22:21.283 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.277+0000 7f32800ff640 1 librados: calling monclient init 2024-03-25T18:22:21.283 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.277+0000 7f32800ff640 10 monclient: init 2024-03-25T18:22:21.283 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.278+0000 7f32800ff640 10 monclient: _reopen_session rank -1 2024-03-25T18:22:21.283 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.278+0000 7f32800ff640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:22:21.283 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.278+0000 7f32800ff640 10 monclient(hunting): picked mon.noname-a con 0x7f32780a98f0 addr v1:172.21.15.97:6789/0 2024-03-25T18:22:21.284 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.278+0000 7f32800ff640 10 monclient(hunting): picked mon.noname-c con 0x7f327813f710 addr v1:172.21.15.97:6790/0 2024-03-25T18:22:21.284 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.278+0000 7f32800ff640 10 monclient(hunting): picked mon.noname-b con 0x7f32781430c0 addr v1:172.21.15.98:6789/0 2024-03-25T18:22:21.284 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.278+0000 7f32800ff640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3278137440 con 0x7f32780a98f0 2024-03-25T18:22:21.284 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.278+0000 7f32800ff640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3278137690 con 0x7f32781430c0 2024-03-25T18:22:21.284 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.278+0000 7f32800ff640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32781d7ca0 con 0x7f327813f710 2024-03-25T18:22:21.284 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.278+0000 7f32800ff640 10 monclient(hunting): _renew_subs 2024-03-25T18:22:21.284 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.278+0000 7f32800ff640 10 monclient(hunting): authenticate will time out at 7953.553544s 2024-03-25T18:22:21.290 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.289+0000 7f327d69a640 1 --1- >> v1:172.21.15.97:6790/0 conn(0x7f327813f710 0x7f32781d7590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.97:6790/0 says I am v1:172.21.15.97:47402/0 (socket says 172.21.15.97:47402) 2024-03-25T18:22:21.290 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.290+0000 7f327d69a640 1 -- 172.21.15.97:0/2549349800 learned_addr learned my addr 172.21.15.97:0/2549349800 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:22:21.290 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.290+0000 7f326e7fc640 10 client.?.objecter ms_handle_connect 0x7f327813f710 2024-03-25T18:22:21.290 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.290+0000 7f326e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:21.304 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.303+0000 7f326e7fc640 10 client.?.objecter ms_handle_connect 0x7f32780a98f0 2024-03-25T18:22:21.304 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.303+0000 7f326e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:21.304 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.304+0000 7f326e7fc640 10 client.?.objecter ms_handle_connect 0x7f32781430c0 2024-03-25T18:22:21.304 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.304+0000 7f326e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:21.308 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.307+0000 7f326e7fc640 1 -- 172.21.15.97:0/2549349800 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1097057820 0 0) 0x7f32781d7ca0 con 0x7f327813f710 2024-03-25T18:22:21.308 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.307+0000 7f326e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:22:21.308 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.307+0000 7f326e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:21.308 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.307+0000 7f326e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:21.308 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.307+0000 7f326e7fc640 10 monclient(hunting): my global_id is 4490 2024-03-25T18:22:21.308 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.307+0000 7f326e7fc640 1 -- 172.21.15.97:0/2549349800 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f325c004eb0 con 0x7f327813f710 2024-03-25T18:22:21.313 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.313+0000 7f326e7fc640 1 -- 172.21.15.97:0/2549349800 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2620992552 0 0) 0x7f3278137690 con 0x7f32781430c0 2024-03-25T18:22:21.313 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.313+0000 7f326e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:22:21.314 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.313+0000 7f326e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:21.314 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.313+0000 7f326e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:21.314 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.313+0000 7f326e7fc640 10 monclient(hunting): my global_id is 4498 2024-03-25T18:22:21.314 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.313+0000 7f326e7fc640 1 -- 172.21.15.97:0/2549349800 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32781d7ca0 con 0x7f32781430c0 2024-03-25T18:22:21.319 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.318+0000 7f326e7fc640 1 -- 172.21.15.97:0/2549349800 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2530631026 0 0) 0x7f3278137440 con 0x7f32780a98f0 2024-03-25T18:22:21.319 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.318+0000 7f326e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:22:21.319 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.318+0000 7f326e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:21.319 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.318+0000 7f326e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:21.319 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.318+0000 7f326e7fc640 10 monclient(hunting): my global_id is 4494 2024-03-25T18:22:21.319 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.318+0000 7f326e7fc640 1 -- 172.21.15.97:0/2549349800 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3278137690 con 0x7f32780a98f0 2024-03-25T18:22:21.327 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.326+0000 7f326e7fc640 1 -- 172.21.15.97:0/2549349800 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 403468064 0 0) 0x7f32781d7ca0 con 0x7f32781430c0 2024-03-25T18:22:21.327 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.326+0000 7f326e7fc640 1 -- 172.21.15.97:0/2549349800 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3278137440 con 0x7f32781430c0 2024-03-25T18:22:21.327 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.326+0000 7f326e7fc640 1 -- 172.21.15.97:0/2549349800 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f3270001e60 con 0x7f32781430c0 2024-03-25T18:22:21.327 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.326+0000 7f326e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:21.327 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.326+0000 7f326e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:22:21.327 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.326+0000 7f326e7fc640 10 monclient(hunting): dump: 2024-03-25T18:22:21.327 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:22:21.327 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:22:21.327 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:22:21.327 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:22:21.328 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:21.328 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:22:21.328 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:22:21.328 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:22:21.328 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:22:21.328 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:22:21.328 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.326+0000 7f326e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:22:21.334 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.333+0000 7f326e7fc640 1 -- 172.21.15.97:0/2549349800 <== mon.2 v1:172.21.15.97:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1375911375 0 0) 0x7f325c004eb0 con 0x7f327813f710 2024-03-25T18:22:21.334 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.333+0000 7f326e7fc640 1 -- 172.21.15.97:0/2549349800 --> v1:172.21.15.97:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f32781d7ca0 con 0x7f327813f710 2024-03-25T18:22:21.334 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.333+0000 7f326e7fc640 1 -- 172.21.15.97:0/2549349800 <== mon.2 v1:172.21.15.97:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f3268001950 con 0x7f327813f710 2024-03-25T18:22:21.334 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.333+0000 7f326e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:21.334 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.333+0000 7f326e7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:22:21.334 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.333+0000 7f326e7fc640 10 monclient(hunting): dump: 2024-03-25T18:22:21.334 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:22:21.334 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:22:21.334 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:22:21.334 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:22:21.335 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:21.335 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:22:21.335 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:22:21.335 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:22:21.335 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:22:21.335 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:22:21.353 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.352+0000 7f326e7fc640 1 -- 172.21.15.97:0/2549349800 <== mon.2 v1:172.21.15.97:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 335533111 0 0) 0x7f32781d7ca0 con 0x7f327813f710 2024-03-25T18:22:21.353 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.352+0000 7f326e7fc640 1 -- 172.21.15.97:0/2549349800 >> v1:172.21.15.98:6789/0 conn(0x7f32781430c0 legacy=0x7f3278132f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:21.353 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.352+0000 7f326e7fc640 1 -- 172.21.15.97:0/2549349800 >> v1:172.21.15.97:6789/0 conn(0x7f32780a98f0 legacy=0x7f3278139400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:21.353 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.352+0000 7f326e7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:22:21.353 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.352+0000 7f326e7fc640 1 monclient: found mon.c 2024-03-25T18:22:21.353 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.352+0000 7f326e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:22:21.353 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.352+0000 7f326e7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.97:6790/0 2024-03-25T18:22:21.353 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.352+0000 7f326e7fc640 1 -- 172.21.15.97:0/2549349800 --> v1:172.21.15.97:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3278136b60 con 0x7f327813f710 2024-03-25T18:22:21.353 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.352+0000 7f326e7fc640 10 monclient: _finish_auth 0 2024-03-25T18:22:21.353 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.352+0000 7f326e7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:22:21.353 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.352+0000 7f326e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:22:21.353 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.352+0000 7f32800ff640 5 monclient: authenticate success, global_id 4490 2024-03-25T18:22:21.354 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.352+0000 7f32800ff640 10 monclient: _renew_subs 2024-03-25T18:22:21.354 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.352+0000 7f32800ff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.97:6790/0 2024-03-25T18:22:21.354 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.352+0000 7f32800ff640 1 -- 172.21.15.97:0/2549349800 --> v1:172.21.15.97:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3278137bb0 con 0x7f327813f710 2024-03-25T18:22:21.354 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.352+0000 7f32800ff640 10 client.4490.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:22:21.354 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.352+0000 7f32800ff640 10 monclient: _renew_subs 2024-03-25T18:22:21.354 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.352+0000 7f32800ff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.97:6790/0 2024-03-25T18:22:21.354 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.352+0000 7f32800ff640 1 -- 172.21.15.97:0/2549349800 --> v1:172.21.15.97:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f32780b7330 con 0x7f327813f710 2024-03-25T18:22:21.354 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.352+0000 7f32800ff640 1 librados: init done 2024-03-25T18:22:21.355 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.353+0000 7f32800ff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:21.355 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.353+0000 7f32800ff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:21.355 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.353+0000 7f32800ff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.97:6790/0 2024-03-25T18:22:21.355 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.353+0000 7f32800ff640 1 -- 172.21.15.97:0/2549349800 --> v1:172.21.15.97:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3278137820 con 0x7f327813f710 2024-03-25T18:22:21.672 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.672+0000 7f326e7fc640 1 -- 172.21.15.97:0/2549349800 <== mon.2 v1:172.21.15.97:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f32680047e0 con 0x7f327813f710 2024-03-25T18:22:21.672 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.672+0000 7f326e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:22:21.672 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.672+0000 7f326e7fc640 1 -- 172.21.15.97:0/2549349800 <== mon.2 v1:172.21.15.97:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f3268004af0 con 0x7f327813f710 2024-03-25T18:22:21.672 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.672+0000 7f326e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:21.673 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.672+0000 7f326e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:22:21.673 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.672+0000 7f326e7fc640 10 monclient: dump: 2024-03-25T18:22:21.673 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:22:21.673 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:22:21.673 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:22:21.673 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:22:21.673 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:21.673 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:22:21.673 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:22:21.673 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:22:21.673 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:22:21.673 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:22:21.673 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.672+0000 7f326ffff640 4 set_mon_vals no callback set 2024-03-25T18:22:21.674 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.673+0000 7f326e7fc640 1 -- 172.21.15.97:0/2549349800 <== mon.2 v1:172.21.15.97:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2377622137 0 0) 0x7f326801aac0 con 0x7f327813f710 2024-03-25T18:22:21.674 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.673+0000 7f327de9b640 10 monclient: get_auth_request con 0x7f325c06e040 auth_method 0 2024-03-25T18:22:21.674 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.674+0000 7f326e7fc640 10 client.4490.objecter ms_handle_connect 0x7f325c06e040 2024-03-25T18:22:21.678 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.677+0000 7f326e7fc640 1 -- 172.21.15.97:0/2549349800 <== mon.2 v1:172.21.15.97:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 430711918 0 0) 0x7f3268054840 con 0x7f327813f710 2024-03-25T18:22:21.678 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.677+0000 7f326e7fc640 10 client.4490.objecter ms_dispatch 0x7f3278000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:22:21.678 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.677+0000 7f326e7fc640 3 client.4490.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:22:21.678 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.677+0000 7f326e7fc640 3 client.4490.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:22:21.678 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.677+0000 7f326e7fc640 20 client.4490.objecter dump_active .. 0 homeless 2024-03-25T18:22:21.678 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.677+0000 7f326e7fc640 1 -- 172.21.15.97:0/2549349800 <== mon.2 v1:172.21.15.97:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3268050180 con 0x7f327813f710 2024-03-25T18:22:21.678 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.677+0000 7f326e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:21.678 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.677+0000 7f326e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:22:21.910 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.909+0000 7f32800ff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:21.910 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.909+0000 7f32800ff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:21.910 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.909+0000 7f32800ff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.97:6790/0 2024-03-25T18:22:21.910 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.909+0000 7f32800ff640 1 -- 172.21.15.97:0/2549349800 --> v1:172.21.15.97:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f32780627b0 con 0x7f327813f710 2024-03-25T18:22:21.964 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.963+0000 7f326e7fc640 1 -- 172.21.15.97:0/2549349800 <== mon.2 v1:172.21.15.97:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2169427412) 0x7f3268052280 con 0x7f327813f710 2024-03-25T18:22:21.964 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.963+0000 7f326e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:21.964 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.963+0000 7f326e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:22:21.964 INFO:teuthology.misc.health.smithi097.stdout: 2024-03-25T18:22:21.965 INFO:teuthology.misc.health.smithi097.stdout:{"epoch":10,"fsid":"4c3d472c-79b8-486b-9e3d-2738d4306754","created":"2024-03-25T18:16:14.737309+0000","modified":"2024-03-25T18:16:20.357492+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:20.357492+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cb5152b9-262f-45c3-94b8-18d0e81df558","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6daf8793-0838-473d-a7d6-9a0d2894c75d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"136f1493-3b9c-4f43-a2e7-641482d69dae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"397d538c-4f64-4daf-9e8b-662d09f91327","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4f1df08-663f-4bda-93f8-3878ab7c24b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c4b18d70-a3ce-4b15-9b70-eae5525bd761","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8dd4033a-de0e-4f8a-96dc-59f3c1f825b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0077722e-ffa6-4a3d-9891-929d9869b286","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:22:21.967 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.966+0000 7f32800ff640 10 librados: watch_flush enter 2024-03-25T18:22:21.967 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.966+0000 7f32800ff640 10 librados: watch_flush exit 2024-03-25T18:22:21.967 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.966+0000 7f32800ff640 20 client.4490.objecter shutdown clearing up homeless session... 2024-03-25T18:22:21.967 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.966+0000 7f32800ff640 10 client.4490.objecter successfully canceled tick 2024-03-25T18:22:21.967 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.966+0000 7f32800ff640 1 -- 172.21.15.97:0/2549349800 >> v1:172.21.15.98:6800/800387573 conn(0x7f325c06e040 legacy=0x7f325c070500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:21.968 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.966+0000 7f32800ff640 10 monclient: shutdown 2024-03-25T18:22:21.968 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.966+0000 7f32800ff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:22:21.968 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.966+0000 7f32800ff640 1 -- 172.21.15.97:0/2549349800 >> v1:172.21.15.97:6790/0 conn(0x7f327813f710 legacy=0x7f32781d7590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:21.968 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.966+0000 7f32800ff640 1 -- 172.21.15.97:0/2549349800 shutdown_connections 2024-03-25T18:22:21.968 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.966+0000 7f32800ff640 1 -- 172.21.15.97:0/2549349800 >> 172.21.15.97:0/2549349800 conn(0x7f32780b7ea0 msgr2=0x7f3278074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:22:21.968 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.966+0000 7f32800ff640 1 -- 172.21.15.97:0/2549349800 shutdown_connections 2024-03-25T18:22:21.968 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.966+0000 7f32800ff640 1 -- 172.21.15.97:0/2549349800 wait complete. 2024-03-25T18:22:21.968 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:21.966+0000 7f32800ff640 1 librados: shutdown 2024-03-25T18:22:21.977 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:22:28.783 DEBUG:teuthology.orchestra.run.smithi097:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:22:28.897 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:28.897+0000 7f789260e640 10 monclient: get_monmap_and_config 2024-03-25T18:22:28.897 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:28.897+0000 7f789260e640 10 monclient: build_initial_monmap 2024-03-25T18:22:28.897 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:28.897+0000 7f789260e640 10 monclient: monmap: 2024-03-25T18:22:28.897 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:22:28.897 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:22:28.897 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:22:28.897291+0000 2024-03-25T18:22:28.897 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:22:28.897291+0000 2024-03-25T18:22:28.897 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:22:28.898 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:22:28.898 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:22:28.898 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T18:22:28.898 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-b 2024-03-25T18:22:28.898 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:22:28.899 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:28.897+0000 7f789260e640 1 Processor -- start 2024-03-25T18:22:28.899 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:28.897+0000 7f789260e640 1 -- start start 2024-03-25T18:22:28.900 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:28.897+0000 7f789260e640 10 monclient: init 2024-03-25T18:22:28.900 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:28.898+0000 7f789260e640 10 monclient: _reopen_session rank -1 2024-03-25T18:22:28.900 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:28.898+0000 7f789260e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:22:28.900 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:28.898+0000 7f789260e640 10 monclient(hunting): picked mon.noname-a con 0x7f788c0b4d60 addr v1:172.21.15.97:6789/0 2024-03-25T18:22:28.900 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:28.898+0000 7f789260e640 10 monclient(hunting): picked mon.noname-c con 0x7f788c0b1730 addr v1:172.21.15.98:6789/0 2024-03-25T18:22:28.900 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:28.898+0000 7f789260e640 10 monclient(hunting): picked mon.noname-b con 0x7f788c14be90 addr v1:172.21.15.97:6790/0 2024-03-25T18:22:28.900 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:28.898+0000 7f789260e640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f788c0af940 con 0x7f788c0b4d60 2024-03-25T18:22:28.900 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:28.898+0000 7f789260e640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f788c0b0a70 con 0x7f788c0b1730 2024-03-25T18:22:28.900 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:28.898+0000 7f789260e640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f788c0b0d50 con 0x7f788c14be90 2024-03-25T18:22:28.900 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:28.898+0000 7f789260e640 10 monclient(hunting): _renew_subs 2024-03-25T18:22:28.900 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:28.898+0000 7f789260e640 10 monclient(hunting): authenticate will time out at 7961.173761s 2024-03-25T18:22:28.907 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:28.906+0000 7f7890bab640 1 --1- >> v1:172.21.15.97:6790/0 conn(0x7f788c14be90 0x7f788c14e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6790/0 says I am v1:172.21.15.97:52164/0 (socket says 172.21.15.97:52164) 2024-03-25T18:22:28.907 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:28.906+0000 7f7890bab640 1 -- 172.21.15.97:0/2719129963 learned_addr learned my addr 172.21.15.97:0/2719129963 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:22:28.917 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:28.917+0000 7f788affd640 1 -- 172.21.15.97:0/2719129963 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1872166732 0 0) 0x7f788c0b0d50 con 0x7f788c14be90 2024-03-25T18:22:28.917 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:28.917+0000 7f788affd640 10 monclient(hunting): _negotiate 2024-03-25T18:22:28.917 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:28.917+0000 7f788affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:28.917 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:28.917+0000 7f788affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:28.917 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:28.917+0000 7f788affd640 10 monclient(hunting): my global_id is 4493 2024-03-25T18:22:28.918 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:28.917+0000 7f788affd640 1 -- 172.21.15.97:0/2719129963 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7874003520 con 0x7f788c14be90 2024-03-25T18:22:28.926 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:28.925+0000 7f788affd640 1 -- 172.21.15.97:0/2719129963 <== mon.2 v1:172.21.15.97:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 212426347 0 0) 0x7f7874003520 con 0x7f788c14be90 2024-03-25T18:22:28.926 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:28.925+0000 7f788affd640 1 -- 172.21.15.97:0/2719129963 >> v1:172.21.15.98:6789/0 conn(0x7f788c0b1730 legacy=0x7f788c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:28.926 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:28.925+0000 7f788affd640 1 -- 172.21.15.97:0/2719129963 >> v1:172.21.15.97:6789/0 conn(0x7f788c0b4d60 legacy=0x7f788c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:28.926 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:28.925+0000 7f788affd640 10 monclient: _finish_hunting 0 2024-03-25T18:22:28.927 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:28.925+0000 7f788affd640 1 monclient: found mon.noname-b 2024-03-25T18:22:28.927 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:28.925+0000 7f788affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:22:28.927 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:28.925+0000 7f788affd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.97:6790/0 2024-03-25T18:22:28.927 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:28.925+0000 7f788affd640 1 -- 172.21.15.97:0/2719129963 --> v1:172.21.15.97:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f788c0b1030 con 0x7f788c14be90 2024-03-25T18:22:28.927 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:28.925+0000 7f788affd640 10 monclient: _finish_auth 0 2024-03-25T18:22:28.927 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:28.925+0000 7f788affd640 10 monclient: _check_auth_tickets 2024-03-25T18:22:28.927 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:28.925+0000 7f788affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:22:28.927 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:28.925+0000 7f788affd640 1 -- 172.21.15.97:0/2719129963 <== mon.2 v1:172.21.15.97:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f7884001a10 con 0x7f788c14be90 2024-03-25T18:22:28.927 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:28.925+0000 7f789260e640 5 monclient: authenticate success, global_id 4493 2024-03-25T18:22:28.927 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:28.925+0000 7f789260e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:22:28.927 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:28.925+0000 7f788affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:28.927 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:28.925+0000 7f788affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:22:28.927 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:28.925+0000 7f788affd640 10 monclient: dump: 2024-03-25T18:22:28.927 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:22:28.928 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:22:28.928 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:22:28.928 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:22:28.928 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:28.928 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:22:28.928 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:22:28.928 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:22:28.928 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:22:28.928 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:22:28.928 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:28.925+0000 7f789260e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:22:28.936 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:28.935+0000 7f788affd640 1 -- 172.21.15.97:0/2719129963 <== mon.2 v1:172.21.15.97:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7884001b70 con 0x7f788c14be90 2024-03-25T18:22:28.936 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:28.935+0000 7f788affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:22:28.936 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:28.935+0000 7f788affd640 1 -- 172.21.15.97:0/2719129963 <== mon.2 v1:172.21.15.97:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f7884001f80 con 0x7f788c14be90 2024-03-25T18:22:28.936 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:28.935+0000 7f789260e640 10 monclient: get_monmap_and_config success 2024-03-25T18:22:28.936 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:28.935+0000 7f789260e640 4 set_mon_vals no callback set 2024-03-25T18:22:28.937 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:28.935+0000 7f788affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:28.937 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:28.936+0000 7f788affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:22:28.937 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:28.936+0000 7f788affd640 10 monclient: dump: 2024-03-25T18:22:28.937 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:22:28.937 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:22:28.937 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:22:28.937 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:22:28.937 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:28.937 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:22:28.937 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:22:28.937 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:22:28.937 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:22:28.937 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:22:28.939 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:28.936+0000 7f789260e640 10 monclient: shutdown 2024-03-25T18:22:28.939 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:28.936+0000 7f789260e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:22:28.939 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:28.936+0000 7f789260e640 1 -- 172.21.15.97:0/2719129963 >> v1:172.21.15.97:6790/0 conn(0x7f788c14be90 legacy=0x7f788c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:28.939 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:28.936+0000 7f789260e640 1 -- 172.21.15.97:0/2719129963 shutdown_connections 2024-03-25T18:22:28.939 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:28.936+0000 7f789260e640 1 -- 172.21.15.97:0/2719129963 >> 172.21.15.97:0/2719129963 conn(0x7f788c0617a0 msgr2=0x7f788c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:22:28.940 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:28.936+0000 7f789260e640 1 -- 172.21.15.97:0/2719129963 shutdown_connections 2024-03-25T18:22:28.940 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:28.936+0000 7f789260e640 1 -- 172.21.15.97:0/2719129963 wait complete. 2024-03-25T18:22:28.940 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:28.937+0000 7f789260e640 10 monclient: build_initial_monmap 2024-03-25T18:22:28.940 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:28.937+0000 7f789260e640 10 monclient: monmap: 2024-03-25T18:22:28.940 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:22:28.940 INFO:teuthology.misc.health.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:22:28.940 INFO:teuthology.misc.health.smithi097.stderr:last_changed 0.000000 2024-03-25T18:22:28.940 INFO:teuthology.misc.health.smithi097.stderr:created 0.000000 2024-03-25T18:22:28.940 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:22:28.940 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 1 2024-03-25T18:22:28.940 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:22:28.940 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T18:22:28.940 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-c 2024-03-25T18:22:28.940 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:22:28.940 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:28.937+0000 7f789260e640 1 librados: starting msgr at 2024-03-25T18:22:28.941 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:28.937+0000 7f789260e640 1 librados: starting objecter 2024-03-25T18:22:28.941 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:28.938+0000 7f789260e640 1 Processor -- start 2024-03-25T18:22:28.941 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:28.938+0000 7f789260e640 1 -- start start 2024-03-25T18:22:28.941 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:28.938+0000 7f789260e640 1 librados: setting wanted keys 2024-03-25T18:22:28.941 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:28.938+0000 7f789260e640 1 librados: calling monclient init 2024-03-25T18:22:28.941 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:28.938+0000 7f789260e640 10 monclient: init 2024-03-25T18:22:28.941 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:28.938+0000 7f789260e640 10 monclient: _reopen_session rank -1 2024-03-25T18:22:28.941 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:28.938+0000 7f789260e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:22:28.941 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:28.939+0000 7f789260e640 10 monclient(hunting): picked mon.noname-b con 0x7f788c0b1730 addr v1:172.21.15.98:6789/0 2024-03-25T18:22:28.941 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:28.939+0000 7f789260e640 10 monclient(hunting): picked mon.noname-a con 0x7f788c0b4d60 addr v1:172.21.15.97:6789/0 2024-03-25T18:22:28.941 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:28.939+0000 7f789260e640 10 monclient(hunting): picked mon.noname-c con 0x7f788c14be90 addr v1:172.21.15.97:6790/0 2024-03-25T18:22:28.941 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:28.939+0000 7f789260e640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f788c137ff0 con 0x7f788c0b4d60 2024-03-25T18:22:28.941 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:28.939+0000 7f789260e640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f788c1366d0 con 0x7f788c0b1730 2024-03-25T18:22:28.941 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:28.939+0000 7f789260e640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f788c137c80 con 0x7f788c14be90 2024-03-25T18:22:28.941 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:28.939+0000 7f789260e640 10 monclient(hunting): _renew_subs 2024-03-25T18:22:28.942 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:28.939+0000 7f789260e640 10 monclient(hunting): authenticate will time out at 7961.214061s 2024-03-25T18:22:28.948 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:28.948+0000 7f788bfff640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f788c0b1730 0x7f788c1d7910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.97:50870/0 (socket says 172.21.15.97:50870) 2024-03-25T18:22:28.949 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:28.948+0000 7f788bfff640 1 -- 172.21.15.97:0/3207366735 learned_addr learned my addr 172.21.15.97:0/3207366735 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:22:28.949 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:28.949+0000 7f7888ff9640 10 client.?.objecter ms_handle_connect 0x7f788c0b1730 2024-03-25T18:22:28.949 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:28.949+0000 7f7888ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:28.953 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:28.953+0000 7f7888ff9640 10 client.?.objecter ms_handle_connect 0x7f788c14be90 2024-03-25T18:22:28.953 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:28.953+0000 7f7888ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:28.956 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:28.955+0000 7f7888ff9640 10 client.?.objecter ms_handle_connect 0x7f788c0b4d60 2024-03-25T18:22:28.956 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:28.955+0000 7f7888ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:28.957 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:28.957+0000 7f7888ff9640 1 -- 172.21.15.97:0/3207366735 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3506920008 0 0) 0x7f788c1366d0 con 0x7f788c0b1730 2024-03-25T18:22:28.957 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:28.957+0000 7f7888ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:22:28.957 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:28.957+0000 7f7888ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:28.957 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:28.957+0000 7f7888ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:28.957 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:28.957+0000 7f7888ff9640 10 monclient(hunting): my global_id is 4504 2024-03-25T18:22:28.957 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:28.957+0000 7f7888ff9640 1 -- 172.21.15.97:0/3207366735 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7878004eb0 con 0x7f788c0b1730 2024-03-25T18:22:28.964 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:28.963+0000 7f7888ff9640 1 -- 172.21.15.97:0/3207366735 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3926089538 0 0) 0x7f788c137c80 con 0x7f788c14be90 2024-03-25T18:22:28.964 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:28.963+0000 7f7888ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:22:28.964 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:28.963+0000 7f7888ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:28.964 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:28.963+0000 7f7888ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:28.964 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:28.963+0000 7f7888ff9640 10 monclient(hunting): my global_id is 4496 2024-03-25T18:22:28.965 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:28.963+0000 7f7888ff9640 1 -- 172.21.15.97:0/3207366735 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f788c1366d0 con 0x7f788c14be90 2024-03-25T18:22:28.976 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:28.976+0000 7f7888ff9640 1 -- 172.21.15.97:0/3207366735 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 903468085 0 0) 0x7f7878004eb0 con 0x7f788c0b1730 2024-03-25T18:22:28.976 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:28.976+0000 7f7888ff9640 1 -- 172.21.15.97:0/3207366735 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f788c137c80 con 0x7f788c0b1730 2024-03-25T18:22:28.977 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:28.976+0000 7f7888ff9640 1 -- 172.21.15.97:0/3207366735 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f787c001a30 con 0x7f788c0b1730 2024-03-25T18:22:28.977 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:28.976+0000 7f7888ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:28.977 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:28.976+0000 7f7888ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:22:28.977 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:28.976+0000 7f7888ff9640 10 monclient(hunting): dump: 2024-03-25T18:22:28.977 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:22:28.977 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:22:28.977 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:22:28.977 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:22:28.977 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:28.977 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:22:28.977 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:22:28.977 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:22:28.977 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:22:28.977 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:22:28.977 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:28.976+0000 7f7888ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:22:28.978 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:28.978+0000 7f7888ff9640 1 -- 172.21.15.97:0/3207366735 <== mon.2 v1:172.21.15.97:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 155798374 0 0) 0x7f788c1366d0 con 0x7f788c14be90 2024-03-25T18:22:28.978 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:28.978+0000 7f7888ff9640 1 -- 172.21.15.97:0/3207366735 --> v1:172.21.15.97:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7878004eb0 con 0x7f788c14be90 2024-03-25T18:22:28.979 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:28.978+0000 7f7888ff9640 1 -- 172.21.15.97:0/3207366735 <== mon.2 v1:172.21.15.97:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f7884001ea0 con 0x7f788c14be90 2024-03-25T18:22:28.979 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:28.978+0000 7f7888ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:28.979 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:28.978+0000 7f7888ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:22:28.979 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:28.978+0000 7f7888ff9640 10 monclient(hunting): dump: 2024-03-25T18:22:28.979 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:22:28.979 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:22:28.979 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:22:28.979 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:22:28.979 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:28.979 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:22:28.979 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:22:28.979 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:22:28.979 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:22:28.979 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:22:28.988 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:28.987+0000 7f7888ff9640 1 -- 172.21.15.97:0/3207366735 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 534035044 0 0) 0x7f788c137ff0 con 0x7f788c0b4d60 2024-03-25T18:22:28.988 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:28.988+0000 7f7888ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:22:28.988 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:28.988+0000 7f7888ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:28.988 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:28.988+0000 7f7888ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:28.988 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:28.988+0000 7f7888ff9640 10 monclient(hunting): my global_id is 4500 2024-03-25T18:22:28.988 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:28.988+0000 7f7888ff9640 1 -- 172.21.15.97:0/3207366735 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f788c1366d0 con 0x7f788c0b4d60 2024-03-25T18:22:28.991 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:28.991+0000 7f7888ff9640 1 -- 172.21.15.97:0/3207366735 <== mon.2 v1:172.21.15.97:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3012655408 0 0) 0x7f7878004eb0 con 0x7f788c14be90 2024-03-25T18:22:28.991 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:28.991+0000 7f7888ff9640 1 -- 172.21.15.97:0/3207366735 >> v1:172.21.15.98:6789/0 conn(0x7f788c0b1730 legacy=0x7f788c1d7910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:28.992 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:28.991+0000 7f7888ff9640 1 -- 172.21.15.97:0/3207366735 >> v1:172.21.15.97:6789/0 conn(0x7f788c0b4d60 legacy=0x7f788c142b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:28.992 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:28.991+0000 7f7888ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:22:28.992 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:28.991+0000 7f7888ff9640 1 monclient: found mon.c 2024-03-25T18:22:28.992 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:28.991+0000 7f7888ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:22:28.992 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:28.991+0000 7f7888ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.97:6790/0 2024-03-25T18:22:28.992 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:28.991+0000 7f7888ff9640 1 -- 172.21.15.97:0/3207366735 --> v1:172.21.15.97:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f788c138480 con 0x7f788c14be90 2024-03-25T18:22:28.992 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:28.991+0000 7f7888ff9640 10 monclient: _finish_auth 0 2024-03-25T18:22:28.992 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:28.991+0000 7f7888ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:22:28.992 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:28.991+0000 7f7888ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:22:28.992 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:28.991+0000 7f789260e640 5 monclient: authenticate success, global_id 4496 2024-03-25T18:22:28.992 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:28.991+0000 7f789260e640 10 monclient: _renew_subs 2024-03-25T18:22:28.992 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:28.991+0000 7f789260e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.97:6790/0 2024-03-25T18:22:28.992 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:28.991+0000 7f789260e640 1 -- 172.21.15.97:0/3207366735 --> v1:172.21.15.97:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f788c1386d0 con 0x7f788c14be90 2024-03-25T18:22:28.992 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:28.991+0000 7f789260e640 10 client.4496.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:22:28.993 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:28.991+0000 7f789260e640 10 monclient: _renew_subs 2024-03-25T18:22:28.993 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:28.991+0000 7f789260e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.97:6790/0 2024-03-25T18:22:28.993 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:28.991+0000 7f789260e640 1 -- 172.21.15.97:0/3207366735 --> v1:172.21.15.97:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f788c141570 con 0x7f788c14be90 2024-03-25T18:22:28.993 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:28.991+0000 7f789260e640 1 librados: init done 2024-03-25T18:22:28.993 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:28.992+0000 7f789260e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:28.993 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:28.992+0000 7f789260e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:28.993 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:28.992+0000 7f789260e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.97:6790/0 2024-03-25T18:22:28.993 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:28.992+0000 7f789260e640 1 -- 172.21.15.97:0/3207366735 --> v1:172.21.15.97:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f788c143250 con 0x7f788c14be90 2024-03-25T18:22:29.287 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:29.286+0000 7f7888ff9640 1 -- 172.21.15.97:0/3207366735 <== mon.2 v1:172.21.15.97:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7884002000 con 0x7f788c14be90 2024-03-25T18:22:29.287 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:29.286+0000 7f7888ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:22:29.287 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:29.286+0000 7f7888ff9640 1 -- 172.21.15.97:0/3207366735 <== mon.2 v1:172.21.15.97:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f7884004e10 con 0x7f788c14be90 2024-03-25T18:22:29.287 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:29.286+0000 7f7888ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:29.287 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:29.286+0000 7f7888ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:22:29.287 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:29.286+0000 7f7888ff9640 10 monclient: dump: 2024-03-25T18:22:29.287 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:22:29.287 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:22:29.287 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:22:29.287 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:22:29.287 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:29.287 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:22:29.288 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:22:29.288 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:22:29.288 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:22:29.288 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:22:29.288 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:29.286+0000 7f788a7fc640 4 set_mon_vals no callback set 2024-03-25T18:22:29.288 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:29.287+0000 7f7888ff9640 1 -- 172.21.15.97:0/3207366735 <== mon.2 v1:172.21.15.97:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2377622137 0 0) 0x7f788401ade0 con 0x7f788c14be90 2024-03-25T18:22:29.289 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:29.288+0000 7f788bfff640 10 monclient: get_auth_request con 0x7f787806df30 auth_method 0 2024-03-25T18:22:29.289 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:29.288+0000 7f7888ff9640 10 client.4496.objecter ms_handle_connect 0x7f787806df30 2024-03-25T18:22:29.293 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:29.292+0000 7f7888ff9640 1 -- 172.21.15.97:0/3207366735 <== mon.2 v1:172.21.15.97:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 430711918 0 0) 0x7f7884054890 con 0x7f788c14be90 2024-03-25T18:22:29.293 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:29.292+0000 7f7888ff9640 10 client.4496.objecter ms_dispatch 0x7f788c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:22:29.293 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:29.292+0000 7f7888ff9640 3 client.4496.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:22:29.293 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:29.292+0000 7f7888ff9640 3 client.4496.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:22:29.293 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:29.292+0000 7f7888ff9640 20 client.4496.objecter dump_active .. 0 homeless 2024-03-25T18:22:29.294 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:29.292+0000 7f7888ff9640 1 -- 172.21.15.97:0/3207366735 <== mon.2 v1:172.21.15.97:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f78840501d0 con 0x7f788c14be90 2024-03-25T18:22:29.294 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:29.293+0000 7f7888ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:29.294 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:29.293+0000 7f7888ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:22:29.517 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:29.516+0000 7f789260e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:29.517 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:29.516+0000 7f789260e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:29.517 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:29.516+0000 7f789260e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.97:6790/0 2024-03-25T18:22:29.517 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:29.516+0000 7f789260e640 1 -- 172.21.15.97:0/3207366735 --> v1:172.21.15.97:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f788c1375f0 con 0x7f788c14be90 2024-03-25T18:22:29.543 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:29.542+0000 7f7888ff9640 1 -- 172.21.15.97:0/3207366735 <== mon.2 v1:172.21.15.97:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2169427412) 0x7f78840522d0 con 0x7f788c14be90 2024-03-25T18:22:29.543 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:29.542+0000 7f7888ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:29.543 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:29.542+0000 7f7888ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:22:29.543 INFO:teuthology.misc.health.smithi097.stdout: 2024-03-25T18:22:29.543 INFO:teuthology.misc.health.smithi097.stdout:{"epoch":10,"fsid":"4c3d472c-79b8-486b-9e3d-2738d4306754","created":"2024-03-25T18:16:14.737309+0000","modified":"2024-03-25T18:16:20.357492+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:20.357492+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cb5152b9-262f-45c3-94b8-18d0e81df558","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6daf8793-0838-473d-a7d6-9a0d2894c75d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"136f1493-3b9c-4f43-a2e7-641482d69dae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"397d538c-4f64-4daf-9e8b-662d09f91327","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4f1df08-663f-4bda-93f8-3878ab7c24b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c4b18d70-a3ce-4b15-9b70-eae5525bd761","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8dd4033a-de0e-4f8a-96dc-59f3c1f825b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0077722e-ffa6-4a3d-9891-929d9869b286","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:22:29.546 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:29.545+0000 7f789260e640 10 librados: watch_flush enter 2024-03-25T18:22:29.546 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:29.545+0000 7f789260e640 10 librados: watch_flush exit 2024-03-25T18:22:29.546 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:29.545+0000 7f789260e640 20 client.4496.objecter shutdown clearing up homeless session... 2024-03-25T18:22:29.546 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:29.545+0000 7f789260e640 10 client.4496.objecter successfully canceled tick 2024-03-25T18:22:29.546 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:29.545+0000 7f789260e640 1 -- 172.21.15.97:0/3207366735 >> v1:172.21.15.98:6800/800387573 conn(0x7f787806df30 legacy=0x7f78780703f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:29.546 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:29.545+0000 7f789260e640 10 monclient: shutdown 2024-03-25T18:22:29.546 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:29.545+0000 7f789260e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:22:29.546 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:29.545+0000 7f789260e640 1 -- 172.21.15.97:0/3207366735 >> v1:172.21.15.97:6790/0 conn(0x7f788c14be90 legacy=0x7f788c135fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:29.546 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:29.545+0000 7f789260e640 1 -- 172.21.15.97:0/3207366735 shutdown_connections 2024-03-25T18:22:29.546 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:29.545+0000 7f789260e640 1 -- 172.21.15.97:0/3207366735 >> 172.21.15.97:0/3207366735 conn(0x7f788c0617a0 msgr2=0x7f788c13c470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:22:29.547 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:29.545+0000 7f789260e640 1 -- 172.21.15.97:0/3207366735 shutdown_connections 2024-03-25T18:22:29.547 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:29.545+0000 7f789260e640 1 -- 172.21.15.97:0/3207366735 wait complete. 2024-03-25T18:22:29.547 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:29.545+0000 7f789260e640 1 librados: shutdown 2024-03-25T18:22:29.558 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:22:33.023 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:06:35.061 34931== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:33.030 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:06:35.069 34931== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:33.064 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:06:35.102 34931== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:33.086 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:06:35.125 34935== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:33.094 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:06:35.133 34935== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:33.097 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:06:35.140 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:33.100 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:06:35.144 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:33.127 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:06:35.166 34935== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:33.130 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:06:35.173 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:36.364 DEBUG:teuthology.orchestra.run.smithi097:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:22:36.480 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.478+0000 7f0a1ae86640 10 monclient: get_monmap_and_config 2024-03-25T18:22:36.480 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.478+0000 7f0a1ae86640 10 monclient: build_initial_monmap 2024-03-25T18:22:36.480 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.478+0000 7f0a1ae86640 10 monclient: monmap: 2024-03-25T18:22:36.480 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:22:36.480 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:22:36.480 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:22:36.480097+0000 2024-03-25T18:22:36.481 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:22:36.480097+0000 2024-03-25T18:22:36.481 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:22:36.481 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:22:36.481 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:22:36.481 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T18:22:36.481 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-b 2024-03-25T18:22:36.481 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:22:36.482 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.479+0000 7f0a1ae86640 1 Processor -- start 2024-03-25T18:22:36.482 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.479+0000 7f0a1ae86640 1 -- start start 2024-03-25T18:22:36.482 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.479+0000 7f0a1ae86640 10 monclient: init 2024-03-25T18:22:36.482 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.480+0000 7f0a1ae86640 10 monclient: _reopen_session rank -1 2024-03-25T18:22:36.483 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.480+0000 7f0a1ae86640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:22:36.483 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.480+0000 7f0a1ae86640 10 monclient(hunting): picked mon.noname-b con 0x7f0a140a98f0 addr v1:172.21.15.97:6790/0 2024-03-25T18:22:36.483 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.480+0000 7f0a1ae86640 10 monclient(hunting): picked mon.noname-a con 0x7f0a1413f710 addr v1:172.21.15.97:6789/0 2024-03-25T18:22:36.483 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.480+0000 7f0a1ae86640 10 monclient(hunting): picked mon.noname-c con 0x7f0a141430c0 addr v1:172.21.15.98:6789/0 2024-03-25T18:22:36.483 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.480+0000 7f0a1ae86640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a14142520 con 0x7f0a1413f710 2024-03-25T18:22:36.483 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.480+0000 7f0a1ae86640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a140b28a0 con 0x7f0a141430c0 2024-03-25T18:22:36.483 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.480+0000 7f0a1ae86640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a140b2a30 con 0x7f0a140a98f0 2024-03-25T18:22:36.483 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.481+0000 7f0a1ae86640 10 monclient(hunting): _renew_subs 2024-03-25T18:22:36.483 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.481+0000 7f0a1ae86640 10 monclient(hunting): authenticate will time out at 7968.756887s 2024-03-25T18:22:36.488 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.487+0000 7f0a19423640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f0a141430c0 0x7f0a1414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.97:52764/0 (socket says 172.21.15.97:52764) 2024-03-25T18:22:36.489 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.487+0000 7f0a19423640 1 -- 172.21.15.97:0/2390039433 learned_addr learned my addr 172.21.15.97:0/2390039433 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:22:36.497 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.496+0000 7f0a137fe640 1 -- 172.21.15.97:0/2390039433 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 568282521 0 0) 0x7f0a140b28a0 con 0x7f0a141430c0 2024-03-25T18:22:36.497 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.496+0000 7f0a137fe640 10 monclient(hunting): _negotiate 2024-03-25T18:22:36.497 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.496+0000 7f0a137fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:36.497 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.496+0000 7f0a137fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:36.497 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.496+0000 7f0a137fe640 10 monclient(hunting): my global_id is 4507 2024-03-25T18:22:36.497 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.496+0000 7f0a137fe640 1 -- 172.21.15.97:0/2390039433 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a04003520 con 0x7f0a141430c0 2024-03-25T18:22:36.509 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.507+0000 7f0a137fe640 1 -- 172.21.15.97:0/2390039433 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2699866064 0 0) 0x7f0a04003520 con 0x7f0a141430c0 2024-03-25T18:22:36.509 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.507+0000 7f0a137fe640 1 -- 172.21.15.97:0/2390039433 >> v1:172.21.15.97:6790/0 conn(0x7f0a140a98f0 legacy=0x7f0a140a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:36.509 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.507+0000 7f0a137fe640 1 -- 172.21.15.97:0/2390039433 >> v1:172.21.15.97:6789/0 conn(0x7f0a1413f710 legacy=0x7f0a141429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:22:36.509 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.507+0000 7f0a137fe640 10 monclient: _finish_hunting 0 2024-03-25T18:22:36.509 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.507+0000 7f0a137fe640 1 monclient: found mon.noname-c 2024-03-25T18:22:36.509 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.507+0000 7f0a137fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:22:36.509 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.507+0000 7f0a137fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.98:6789/0 2024-03-25T18:22:36.509 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.507+0000 7f0a137fe640 1 -- 172.21.15.97:0/2390039433 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a140b44c0 con 0x7f0a141430c0 2024-03-25T18:22:36.509 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.507+0000 7f0a137fe640 10 monclient: _finish_auth 0 2024-03-25T18:22:36.510 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.507+0000 7f0a137fe640 10 monclient: _check_auth_tickets 2024-03-25T18:22:36.510 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.507+0000 7f0a137fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:22:36.510 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.507+0000 7f0a137fe640 1 -- 172.21.15.97:0/2390039433 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f0a0c001a10 con 0x7f0a141430c0 2024-03-25T18:22:36.510 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.507+0000 7f0a137fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:36.510 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.507+0000 7f0a137fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:22:36.510 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.507+0000 7f0a137fe640 10 monclient: dump: 2024-03-25T18:22:36.510 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:22:36.510 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:22:36.510 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:22:36.510 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:22:36.510 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:36.510 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:22:36.510 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:22:36.510 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:22:36.510 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:22:36.511 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:22:36.511 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.507+0000 7f0a1ae86640 5 monclient: authenticate success, global_id 4507 2024-03-25T18:22:36.511 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.507+0000 7f0a1ae86640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:22:36.512 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.511+0000 7f0a137fe640 1 -- 172.21.15.97:0/2390039433 <== mon.1 v1:172.21.15.98:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0a0c001be0 con 0x7f0a141430c0 2024-03-25T18:22:36.512 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.511+0000 7f0a137fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:22:36.512 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.511+0000 7f0a1ae86640 10 monclient: get_monmap_and_config success 2024-03-25T18:22:36.513 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.511+0000 7f0a1ae86640 4 set_mon_vals no callback set 2024-03-25T18:22:36.514 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.511+0000 7f0a137fe640 1 -- 172.21.15.97:0/2390039433 <== mon.1 v1:172.21.15.98:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f0a0c002030 con 0x7f0a141430c0 2024-03-25T18:22:36.514 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.511+0000 7f0a137fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:36.514 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.511+0000 7f0a137fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:22:36.514 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.511+0000 7f0a137fe640 10 monclient: dump: 2024-03-25T18:22:36.514 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:22:36.514 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:22:36.514 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:22:36.515 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:22:36.515 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:36.515 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:22:36.515 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:22:36.515 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:22:36.515 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:22:36.515 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:22:36.515 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.512+0000 7f0a1ae86640 10 monclient: shutdown 2024-03-25T18:22:36.515 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.512+0000 7f0a1ae86640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:22:36.515 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.512+0000 7f0a1ae86640 1 -- 172.21.15.97:0/2390039433 >> v1:172.21.15.98:6789/0 conn(0x7f0a141430c0 legacy=0x7f0a1414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:36.515 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.512+0000 7f0a1ae86640 1 -- 172.21.15.97:0/2390039433 shutdown_connections 2024-03-25T18:22:36.515 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.512+0000 7f0a1ae86640 1 -- 172.21.15.97:0/2390039433 >> 172.21.15.97:0/2390039433 conn(0x7f0a140b7ea0 msgr2=0x7f0a140b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:22:36.516 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.512+0000 7f0a1ae86640 1 -- 172.21.15.97:0/2390039433 shutdown_connections 2024-03-25T18:22:36.516 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.512+0000 7f0a1ae86640 1 -- 172.21.15.97:0/2390039433 wait complete. 2024-03-25T18:22:36.517 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.513+0000 7f0a1ae86640 10 monclient: build_initial_monmap 2024-03-25T18:22:36.517 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.513+0000 7f0a1ae86640 10 monclient: monmap: 2024-03-25T18:22:36.517 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:22:36.517 INFO:teuthology.misc.health.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:22:36.517 INFO:teuthology.misc.health.smithi097.stderr:last_changed 0.000000 2024-03-25T18:22:36.517 INFO:teuthology.misc.health.smithi097.stderr:created 0.000000 2024-03-25T18:22:36.518 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:22:36.518 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 1 2024-03-25T18:22:36.518 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:22:36.518 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T18:22:36.518 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-c 2024-03-25T18:22:36.518 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:22:36.518 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.513+0000 7f0a1ae86640 1 librados: starting msgr at 2024-03-25T18:22:36.518 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.513+0000 7f0a1ae86640 1 librados: starting objecter 2024-03-25T18:22:36.518 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.513+0000 7f0a1ae86640 1 Processor -- start 2024-03-25T18:22:36.518 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.514+0000 7f0a1ae86640 1 -- start start 2024-03-25T18:22:36.518 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.514+0000 7f0a1ae86640 1 librados: setting wanted keys 2024-03-25T18:22:36.518 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.514+0000 7f0a1ae86640 1 librados: calling monclient init 2024-03-25T18:22:36.518 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.514+0000 7f0a1ae86640 10 monclient: init 2024-03-25T18:22:36.519 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.514+0000 7f0a1ae86640 10 monclient: _reopen_session rank -1 2024-03-25T18:22:36.519 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.514+0000 7f0a1ae86640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:22:36.519 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.514+0000 7f0a1ae86640 10 monclient(hunting): picked mon.noname-b con 0x7f0a140a98f0 addr v1:172.21.15.98:6789/0 2024-03-25T18:22:36.519 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.514+0000 7f0a1ae86640 10 monclient(hunting): picked mon.noname-a con 0x7f0a1413f710 addr v1:172.21.15.97:6789/0 2024-03-25T18:22:36.519 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.514+0000 7f0a1ae86640 10 monclient(hunting): picked mon.noname-c con 0x7f0a141430c0 addr v1:172.21.15.97:6790/0 2024-03-25T18:22:36.519 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.514+0000 7f0a1ae86640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a141d7600 con 0x7f0a1413f710 2024-03-25T18:22:36.519 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.514+0000 7f0a1ae86640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a141d51f0 con 0x7f0a140a98f0 2024-03-25T18:22:36.519 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.514+0000 7f0a1ae86640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a141d8480 con 0x7f0a141430c0 2024-03-25T18:22:36.519 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.514+0000 7f0a1ae86640 10 monclient(hunting): _renew_subs 2024-03-25T18:22:36.519 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.514+0000 7f0a1ae86640 10 monclient(hunting): authenticate will time out at 7968.790841s 2024-03-25T18:22:36.526 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.525+0000 7f0a13fff640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f0a1413f710 0x7f0a14138770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:37048/0 (socket says 172.21.15.97:37048) 2024-03-25T18:22:36.526 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.525+0000 7f0a18c22640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f0a140a98f0 0x7f0a141d7d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.97:52812/0 (socket says 172.21.15.97:52812) 2024-03-25T18:22:36.526 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.525+0000 7f0a13fff640 1 -- 172.21.15.97:0/2737496865 learned_addr learned my addr 172.21.15.97:0/2737496865 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:22:36.528 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.526+0000 7f0a117fa640 10 client.?.objecter ms_handle_connect 0x7f0a140a98f0 2024-03-25T18:22:36.528 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.526+0000 7f0a117fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:36.529 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.527+0000 7f0a117fa640 10 client.?.objecter ms_handle_connect 0x7f0a1413f710 2024-03-25T18:22:36.529 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.527+0000 7f0a117fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:36.529 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.527+0000 7f0a117fa640 10 client.?.objecter ms_handle_connect 0x7f0a141430c0 2024-03-25T18:22:36.529 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.527+0000 7f0a117fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:36.535 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.534+0000 7f0a117fa640 1 -- 172.21.15.97:0/2737496865 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2401005680 0 0) 0x7f0a141d8480 con 0x7f0a141430c0 2024-03-25T18:22:36.535 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.534+0000 7f0a117fa640 10 monclient(hunting): _negotiate 2024-03-25T18:22:36.536 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.534+0000 7f0a117fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:36.536 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.534+0000 7f0a117fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:36.536 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.534+0000 7f0a117fa640 10 monclient(hunting): my global_id is 4502 2024-03-25T18:22:36.536 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.534+0000 7f0a117fa640 1 -- 172.21.15.97:0/2737496865 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09f8004eb0 con 0x7f0a141430c0 2024-03-25T18:22:36.539 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.537+0000 7f0a117fa640 1 -- 172.21.15.97:0/2737496865 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4054601529 0 0) 0x7f0a141d51f0 con 0x7f0a140a98f0 2024-03-25T18:22:36.539 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.537+0000 7f0a117fa640 10 monclient(hunting): _negotiate 2024-03-25T18:22:36.539 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.537+0000 7f0a117fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:36.539 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.537+0000 7f0a117fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:36.539 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.537+0000 7f0a117fa640 10 monclient(hunting): my global_id is 4510 2024-03-25T18:22:36.539 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.537+0000 7f0a117fa640 1 -- 172.21.15.97:0/2737496865 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a141d8480 con 0x7f0a140a98f0 2024-03-25T18:22:36.540 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.539+0000 7f0a117fa640 1 -- 172.21.15.97:0/2737496865 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3161179882 0 0) 0x7f0a141d7600 con 0x7f0a1413f710 2024-03-25T18:22:36.540 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.539+0000 7f0a117fa640 10 monclient(hunting): _negotiate 2024-03-25T18:22:36.540 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.539+0000 7f0a117fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:36.540 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.539+0000 7f0a117fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:36.541 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.539+0000 7f0a117fa640 10 monclient(hunting): my global_id is 4503 2024-03-25T18:22:36.541 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.539+0000 7f0a117fa640 1 -- 172.21.15.97:0/2737496865 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a141d51f0 con 0x7f0a1413f710 2024-03-25T18:22:36.550 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.548+0000 7f0a117fa640 1 -- 172.21.15.97:0/2737496865 <== mon.2 v1:172.21.15.97:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 187106299 0 0) 0x7f09f8004eb0 con 0x7f0a141430c0 2024-03-25T18:22:36.550 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.548+0000 7f0a117fa640 1 -- 172.21.15.97:0/2737496865 --> v1:172.21.15.97:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0a141d7600 con 0x7f0a141430c0 2024-03-25T18:22:36.550 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.548+0000 7f0a117fa640 1 -- 172.21.15.97:0/2737496865 <== mon.2 v1:172.21.15.97:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f0a0c001ef0 con 0x7f0a141430c0 2024-03-25T18:22:36.550 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.548+0000 7f0a117fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:36.550 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.548+0000 7f0a117fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:22:36.550 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.548+0000 7f0a117fa640 10 monclient(hunting): dump: 2024-03-25T18:22:36.550 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:22:36.550 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:22:36.550 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:22:36.550 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:22:36.550 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:36.551 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:22:36.551 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:22:36.551 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:22:36.551 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:22:36.551 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:22:36.551 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.548+0000 7f0a117fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:22:36.559 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.557+0000 7f0a117fa640 1 -- 172.21.15.97:0/2737496865 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2847321188 0 0) 0x7f0a141d8480 con 0x7f0a140a98f0 2024-03-25T18:22:36.559 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.557+0000 7f0a117fa640 1 -- 172.21.15.97:0/2737496865 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f09f8004eb0 con 0x7f0a140a98f0 2024-03-25T18:22:36.559 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.557+0000 7f0a117fa640 1 -- 172.21.15.97:0/2737496865 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f0a080018f0 con 0x7f0a140a98f0 2024-03-25T18:22:36.559 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.557+0000 7f0a117fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:36.559 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.557+0000 7f0a117fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:22:36.559 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.557+0000 7f0a117fa640 10 monclient(hunting): dump: 2024-03-25T18:22:36.559 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:22:36.559 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:22:36.559 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:22:36.559 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:22:36.560 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:36.560 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:22:36.560 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:22:36.560 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:22:36.560 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:22:36.560 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:22:36.566 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.564+0000 7f0a117fa640 1 -- 172.21.15.97:0/2737496865 <== mon.1 v1:172.21.15.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 830972820 0 0) 0x7f09f8004eb0 con 0x7f0a140a98f0 2024-03-25T18:22:36.566 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.564+0000 7f0a117fa640 1 -- 172.21.15.97:0/2737496865 >> v1:172.21.15.97:6790/0 conn(0x7f0a141430c0 legacy=0x7f0a141e1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:36.566 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.564+0000 7f0a117fa640 1 -- 172.21.15.97:0/2737496865 >> v1:172.21.15.97:6789/0 conn(0x7f0a1413f710 legacy=0x7f0a14138770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:36.566 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.564+0000 7f0a117fa640 10 monclient: _finish_hunting 0 2024-03-25T18:22:36.567 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.564+0000 7f0a117fa640 1 monclient: found mon.b 2024-03-25T18:22:36.567 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.564+0000 7f0a117fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:22:36.567 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.564+0000 7f0a117fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:22:36.567 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.564+0000 7f0a117fa640 1 -- 172.21.15.97:0/2737496865 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a141d7850 con 0x7f0a140a98f0 2024-03-25T18:22:36.567 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.564+0000 7f0a117fa640 10 monclient: _finish_auth 0 2024-03-25T18:22:36.567 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.564+0000 7f0a117fa640 10 monclient: _check_auth_tickets 2024-03-25T18:22:36.567 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.564+0000 7f0a117fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:22:36.567 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.564+0000 7f0a1ae86640 5 monclient: authenticate success, global_id 4510 2024-03-25T18:22:36.567 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.564+0000 7f0a1ae86640 10 monclient: _renew_subs 2024-03-25T18:22:36.567 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.564+0000 7f0a1ae86640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:22:36.567 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.565+0000 7f0a1ae86640 1 -- 172.21.15.97:0/2737496865 --> v1:172.21.15.98:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a14138e80 con 0x7f0a140a98f0 2024-03-25T18:22:36.567 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.565+0000 7f0a1ae86640 10 client.4510.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:22:36.567 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.565+0000 7f0a1ae86640 10 monclient: _renew_subs 2024-03-25T18:22:36.567 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.565+0000 7f0a1ae86640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:22:36.567 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.565+0000 7f0a1ae86640 1 -- 172.21.15.97:0/2737496865 --> v1:172.21.15.98:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a14138190 con 0x7f0a140a98f0 2024-03-25T18:22:36.567 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.565+0000 7f0a1ae86640 1 librados: init done 2024-03-25T18:22:36.568 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.566+0000 7f0a1ae86640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:36.568 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.566+0000 7f0a1ae86640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:36.568 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.566+0000 7f0a1ae86640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:22:36.568 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.566+0000 7f0a1ae86640 1 -- 172.21.15.97:0/2737496865 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a140abe10 con 0x7f0a140a98f0 2024-03-25T18:22:36.842 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.840+0000 7f0a117fa640 1 -- 172.21.15.97:0/2737496865 <== mon.1 v1:172.21.15.98:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0a080035f0 con 0x7f0a140a98f0 2024-03-25T18:22:36.842 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.840+0000 7f0a117fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:22:36.842 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.841+0000 7f0a117fa640 1 -- 172.21.15.97:0/2737496865 <== mon.1 v1:172.21.15.98:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f0a080039c0 con 0x7f0a140a98f0 2024-03-25T18:22:36.843 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.841+0000 7f0a117fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:36.843 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.841+0000 7f0a117fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:22:36.843 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.841+0000 7f0a117fa640 10 monclient: dump: 2024-03-25T18:22:36.843 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:22:36.843 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:22:36.843 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:22:36.843 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:22:36.843 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:36.843 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:22:36.843 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:22:36.843 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:22:36.843 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:22:36.843 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:22:36.843 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.841+0000 7f0a12ffd640 4 set_mon_vals no callback set 2024-03-25T18:22:36.844 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.842+0000 7f0a117fa640 1 -- 172.21.15.97:0/2737496865 <== mon.1 v1:172.21.15.98:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2377622137 0 0) 0x7f0a08003be0 con 0x7f0a140a98f0 2024-03-25T18:22:36.844 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.842+0000 7f0a13fff640 10 monclient: get_auth_request con 0x7f09f806df60 auth_method 0 2024-03-25T18:22:36.844 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.842+0000 7f0a117fa640 10 client.4510.objecter ms_handle_connect 0x7f09f806df60 2024-03-25T18:22:36.848 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.846+0000 7f0a117fa640 1 -- 172.21.15.97:0/2737496865 <== mon.1 v1:172.21.15.98:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 430711918 0 0) 0x7f0a08054790 con 0x7f0a140a98f0 2024-03-25T18:22:36.848 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.846+0000 7f0a117fa640 10 client.4510.objecter ms_dispatch 0x7f0a14000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:22:36.848 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.846+0000 7f0a117fa640 3 client.4510.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:22:36.848 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.846+0000 7f0a117fa640 3 client.4510.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:22:36.848 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.846+0000 7f0a117fa640 20 client.4510.objecter dump_active .. 0 homeless 2024-03-25T18:22:36.848 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.847+0000 7f0a117fa640 1 -- 172.21.15.97:0/2737496865 <== mon.1 v1:172.21.15.98:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0a080502a0 con 0x7f0a140a98f0 2024-03-25T18:22:36.849 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.847+0000 7f0a117fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:36.849 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:36.847+0000 7f0a117fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:22:37.118 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:37.116+0000 7f0a1ae86640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:37.118 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:37.116+0000 7f0a1ae86640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:37.118 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:37.116+0000 7f0a1ae86640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:22:37.118 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:37.116+0000 7f0a1ae86640 1 -- 172.21.15.97:0/2737496865 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0a140b3ef0 con 0x7f0a140a98f0 2024-03-25T18:22:37.146 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:37.145+0000 7f0a117fa640 1 -- 172.21.15.97:0/2737496865 <== mon.1 v1:172.21.15.98:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2169427412) 0x7f0a080521d0 con 0x7f0a140a98f0 2024-03-25T18:22:37.147 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:37.145+0000 7f0a117fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:37.147 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:37.145+0000 7f0a117fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:22:37.147 INFO:teuthology.misc.health.smithi097.stdout: 2024-03-25T18:22:37.147 INFO:teuthology.misc.health.smithi097.stdout:{"epoch":10,"fsid":"4c3d472c-79b8-486b-9e3d-2738d4306754","created":"2024-03-25T18:16:14.737309+0000","modified":"2024-03-25T18:16:20.357492+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:20.357492+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cb5152b9-262f-45c3-94b8-18d0e81df558","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6daf8793-0838-473d-a7d6-9a0d2894c75d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"136f1493-3b9c-4f43-a2e7-641482d69dae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"397d538c-4f64-4daf-9e8b-662d09f91327","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4f1df08-663f-4bda-93f8-3878ab7c24b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c4b18d70-a3ce-4b15-9b70-eae5525bd761","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8dd4033a-de0e-4f8a-96dc-59f3c1f825b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0077722e-ffa6-4a3d-9891-929d9869b286","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:22:37.149 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:37.147+0000 7f0a1ae86640 10 librados: watch_flush enter 2024-03-25T18:22:37.149 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:37.147+0000 7f0a1ae86640 10 librados: watch_flush exit 2024-03-25T18:22:37.149 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:37.147+0000 7f0a1ae86640 20 client.4510.objecter shutdown clearing up homeless session... 2024-03-25T18:22:37.149 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:37.147+0000 7f0a1ae86640 10 client.4510.objecter successfully canceled tick 2024-03-25T18:22:37.149 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:37.147+0000 7f0a1ae86640 1 -- 172.21.15.97:0/2737496865 >> v1:172.21.15.98:6800/800387573 conn(0x7f09f806df60 legacy=0x7f09f8070420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:37.150 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:37.147+0000 7f0a1ae86640 10 monclient: shutdown 2024-03-25T18:22:37.150 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:37.147+0000 7f0a1ae86640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:22:37.150 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:37.147+0000 7f0a1ae86640 1 -- 172.21.15.97:0/2737496865 >> v1:172.21.15.98:6789/0 conn(0x7f0a140a98f0 legacy=0x7f0a141d7d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:37.150 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:37.148+0000 7f0a1ae86640 1 -- 172.21.15.97:0/2737496865 shutdown_connections 2024-03-25T18:22:37.150 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:37.148+0000 7f0a1ae86640 1 -- 172.21.15.97:0/2737496865 >> 172.21.15.97:0/2737496865 conn(0x7f0a140b7ea0 msgr2=0x7f0a14074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:22:37.150 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:37.148+0000 7f0a1ae86640 1 -- 172.21.15.97:0/2737496865 shutdown_connections 2024-03-25T18:22:37.150 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:37.148+0000 7f0a1ae86640 1 -- 172.21.15.97:0/2737496865 wait complete. 2024-03-25T18:22:37.150 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:37.148+0000 7f0a1ae86640 1 librados: shutdown 2024-03-25T18:22:37.162 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:22:43.968 DEBUG:teuthology.orchestra.run.smithi097:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:22:44.090 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.088+0000 7ff165c86640 10 monclient: get_monmap_and_config 2024-03-25T18:22:44.090 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.088+0000 7ff165c86640 10 monclient: build_initial_monmap 2024-03-25T18:22:44.090 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.088+0000 7ff165c86640 10 monclient: monmap: 2024-03-25T18:22:44.090 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:22:44.090 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:22:44.090 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:22:44.089837+0000 2024-03-25T18:22:44.090 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:22:44.089837+0000 2024-03-25T18:22:44.090 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:22:44.090 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:22:44.091 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:22:44.091 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T18:22:44.091 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-b 2024-03-25T18:22:44.091 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:22:44.092 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.089+0000 7ff165c86640 1 Processor -- start 2024-03-25T18:22:44.092 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.089+0000 7ff165c86640 1 -- start start 2024-03-25T18:22:44.092 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.089+0000 7ff165c86640 10 monclient: init 2024-03-25T18:22:44.092 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.090+0000 7ff165c86640 10 monclient: _reopen_session rank -1 2024-03-25T18:22:44.092 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.090+0000 7ff165c86640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:22:44.092 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.090+0000 7ff165c86640 10 monclient(hunting): picked mon.noname-c con 0x7ff160061890 addr v1:172.21.15.98:6789/0 2024-03-25T18:22:44.092 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.090+0000 7ff165c86640 10 monclient(hunting): picked mon.noname-b con 0x7ff16006e740 addr v1:172.21.15.97:6790/0 2024-03-25T18:22:44.093 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.090+0000 7ff165c86640 10 monclient(hunting): picked mon.noname-a con 0x7ff1600ae280 addr v1:172.21.15.97:6789/0 2024-03-25T18:22:44.093 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.090+0000 7ff165c86640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1600ae830 con 0x7ff1600ae280 2024-03-25T18:22:44.093 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.090+0000 7ff165c86640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff16006f720 con 0x7ff160061890 2024-03-25T18:22:44.093 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.090+0000 7ff165c86640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1600ada80 con 0x7ff16006e740 2024-03-25T18:22:44.093 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.090+0000 7ff165c86640 10 monclient(hunting): _renew_subs 2024-03-25T18:22:44.093 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.090+0000 7ff165c86640 10 monclient(hunting): authenticate will time out at 7976.366546s 2024-03-25T18:22:44.100 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.099+0000 7ff15ffff640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7ff1600ae280 0x7ff160145f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:47858/0 (socket says 172.21.15.97:47858) 2024-03-25T18:22:44.100 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.099+0000 7ff15ffff640 1 -- 172.21.15.97:0/131561877 learned_addr learned my addr 172.21.15.97:0/131561877 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:22:44.119 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.118+0000 7ff15e7fc640 1 -- 172.21.15.97:0/131561877 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 864319928 0 0) 0x7ff16006f720 con 0x7ff160061890 2024-03-25T18:22:44.119 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.118+0000 7ff15e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:22:44.119 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.118+0000 7ff15e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:44.119 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.118+0000 7ff15e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:44.120 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.118+0000 7ff15e7fc640 10 monclient(hunting): my global_id is 4513 2024-03-25T18:22:44.120 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.118+0000 7ff15e7fc640 1 -- 172.21.15.97:0/131561877 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff148003520 con 0x7ff160061890 2024-03-25T18:22:44.121 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.120+0000 7ff15e7fc640 1 -- 172.21.15.97:0/131561877 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2921564804 0 0) 0x7ff1600ada80 con 0x7ff16006e740 2024-03-25T18:22:44.121 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.120+0000 7ff15e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:22:44.121 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.120+0000 7ff15e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:44.121 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.120+0000 7ff15e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:44.122 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.120+0000 7ff15e7fc640 10 monclient(hunting): my global_id is 4505 2024-03-25T18:22:44.122 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.120+0000 7ff15e7fc640 1 -- 172.21.15.97:0/131561877 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff16006f720 con 0x7ff16006e740 2024-03-25T18:22:44.131 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.129+0000 7ff15e7fc640 1 -- 172.21.15.97:0/131561877 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1402539001 0 0) 0x7ff148003520 con 0x7ff160061890 2024-03-25T18:22:44.131 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.129+0000 7ff15e7fc640 1 -- 172.21.15.97:0/131561877 >> v1:172.21.15.97:6790/0 conn(0x7ff16006e740 legacy=0x7ff160145820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:44.131 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.129+0000 7ff15e7fc640 1 -- 172.21.15.97:0/131561877 >> v1:172.21.15.97:6789/0 conn(0x7ff1600ae280 legacy=0x7ff160145f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:44.131 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.129+0000 7ff15e7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:22:44.131 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.129+0000 7ff15e7fc640 1 monclient: found mon.noname-c 2024-03-25T18:22:44.131 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.129+0000 7ff15e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:22:44.131 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.129+0000 7ff15e7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.98:6789/0 2024-03-25T18:22:44.131 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.129+0000 7ff15e7fc640 1 -- 172.21.15.97:0/131561877 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff16013fbd0 con 0x7ff160061890 2024-03-25T18:22:44.131 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.129+0000 7ff15e7fc640 10 monclient: _finish_auth 0 2024-03-25T18:22:44.131 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.129+0000 7ff15e7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:22:44.131 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.129+0000 7ff15e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:22:44.132 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.129+0000 7ff15e7fc640 1 -- 172.21.15.97:0/131561877 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7ff150001690 con 0x7ff160061890 2024-03-25T18:22:44.132 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.129+0000 7ff15e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:44.132 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.129+0000 7ff15e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:22:44.132 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.129+0000 7ff15e7fc640 10 monclient: dump: 2024-03-25T18:22:44.132 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:22:44.132 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:22:44.132 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:22:44.132 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:22:44.132 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:44.132 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:22:44.132 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:22:44.132 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:22:44.132 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:22:44.132 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:22:44.132 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.129+0000 7ff165c86640 5 monclient: authenticate success, global_id 4513 2024-03-25T18:22:44.132 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.129+0000 7ff165c86640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:22:44.134 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.133+0000 7ff15e7fc640 1 -- 172.21.15.97:0/131561877 <== mon.1 v1:172.21.15.98:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff1500017f0 con 0x7ff160061890 2024-03-25T18:22:44.134 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.133+0000 7ff15e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:22:44.134 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.133+0000 7ff15e7fc640 1 -- 172.21.15.97:0/131561877 <== mon.1 v1:172.21.15.98:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7ff150001c00 con 0x7ff160061890 2024-03-25T18:22:44.134 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.133+0000 7ff15e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:44.134 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.133+0000 7ff15e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:22:44.135 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.133+0000 7ff15e7fc640 10 monclient: dump: 2024-03-25T18:22:44.135 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:22:44.135 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:22:44.135 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:22:44.135 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:22:44.135 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:44.135 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:22:44.135 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:22:44.135 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:22:44.135 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:22:44.135 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:22:44.135 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.133+0000 7ff165c86640 10 monclient: get_monmap_and_config success 2024-03-25T18:22:44.135 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.133+0000 7ff165c86640 4 set_mon_vals no callback set 2024-03-25T18:22:44.137 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.133+0000 7ff165c86640 10 monclient: shutdown 2024-03-25T18:22:44.137 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.134+0000 7ff165c86640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:22:44.137 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.134+0000 7ff165c86640 1 -- 172.21.15.97:0/131561877 >> v1:172.21.15.98:6789/0 conn(0x7ff160061890 legacy=0x7ff16013f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:44.137 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.134+0000 7ff165c86640 1 -- 172.21.15.97:0/131561877 shutdown_connections 2024-03-25T18:22:44.137 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.134+0000 7ff165c86640 1 -- 172.21.15.97:0/131561877 >> 172.21.15.97:0/131561877 conn(0x7ff1600b0ef0 msgr2=0x7ff1600b1300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:22:44.137 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.134+0000 7ff165c86640 1 -- 172.21.15.97:0/131561877 shutdown_connections 2024-03-25T18:22:44.137 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.134+0000 7ff165c86640 1 -- 172.21.15.97:0/131561877 wait complete. 2024-03-25T18:22:44.137 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.135+0000 7ff165c86640 10 monclient: build_initial_monmap 2024-03-25T18:22:44.137 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.135+0000 7ff165c86640 10 monclient: monmap: 2024-03-25T18:22:44.137 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:22:44.137 INFO:teuthology.misc.health.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:22:44.137 INFO:teuthology.misc.health.smithi097.stderr:last_changed 0.000000 2024-03-25T18:22:44.137 INFO:teuthology.misc.health.smithi097.stderr:created 0.000000 2024-03-25T18:22:44.138 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:22:44.138 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 1 2024-03-25T18:22:44.138 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:22:44.138 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T18:22:44.138 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-c 2024-03-25T18:22:44.138 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:22:44.138 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.135+0000 7ff165c86640 1 librados: starting msgr at 2024-03-25T18:22:44.138 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.135+0000 7ff165c86640 1 librados: starting objecter 2024-03-25T18:22:44.138 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.135+0000 7ff165c86640 1 Processor -- start 2024-03-25T18:22:44.138 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.135+0000 7ff165c86640 1 -- start start 2024-03-25T18:22:44.138 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.135+0000 7ff165c86640 1 librados: setting wanted keys 2024-03-25T18:22:44.138 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.135+0000 7ff165c86640 1 librados: calling monclient init 2024-03-25T18:22:44.138 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.135+0000 7ff165c86640 10 monclient: init 2024-03-25T18:22:44.139 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.136+0000 7ff165c86640 10 monclient: _reopen_session rank -1 2024-03-25T18:22:44.139 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.136+0000 7ff165c86640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:22:44.140 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.136+0000 7ff165c86640 10 monclient(hunting): picked mon.noname-c con 0x7ff160061890 addr v1:172.21.15.97:6790/0 2024-03-25T18:22:44.140 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.136+0000 7ff165c86640 10 monclient(hunting): picked mon.noname-b con 0x7ff1600ae280 addr v1:172.21.15.98:6789/0 2024-03-25T18:22:44.140 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.136+0000 7ff165c86640 10 monclient(hunting): picked mon.noname-a con 0x7ff160142410 addr v1:172.21.15.97:6789/0 2024-03-25T18:22:44.140 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.136+0000 7ff165c86640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1601dbe00 con 0x7ff160142410 2024-03-25T18:22:44.140 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.136+0000 7ff165c86640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff160135a80 con 0x7ff1600ae280 2024-03-25T18:22:44.140 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.136+0000 7ff165c86640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff160135d60 con 0x7ff160061890 2024-03-25T18:22:44.140 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.136+0000 7ff165c86640 10 monclient(hunting): _renew_subs 2024-03-25T18:22:44.140 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.136+0000 7ff165c86640 10 monclient(hunting): authenticate will time out at 7976.412584s 2024-03-25T18:22:44.143 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.142+0000 7ff15effd640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7ff1600ae280 0x7ff1601dafe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.97:51660/0 (socket says 172.21.15.97:51660) 2024-03-25T18:22:44.143 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.142+0000 7ff15effd640 1 -- 172.21.15.97:0/642759643 learned_addr learned my addr 172.21.15.97:0/642759643 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:22:44.154 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.152+0000 7ff147fff640 10 client.?.objecter ms_handle_connect 0x7ff160142410 2024-03-25T18:22:44.154 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.152+0000 7ff147fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:44.157 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.156+0000 7ff147fff640 10 client.?.objecter ms_handle_connect 0x7ff1600ae280 2024-03-25T18:22:44.157 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.156+0000 7ff147fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:44.164 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.162+0000 7ff147fff640 1 -- 172.21.15.97:0/642759643 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2578205830 0 0) 0x7ff1601dbe00 con 0x7ff160142410 2024-03-25T18:22:44.164 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.162+0000 7ff147fff640 10 monclient(hunting): _negotiate 2024-03-25T18:22:44.164 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.162+0000 7ff147fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:44.164 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.162+0000 7ff147fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:44.164 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.162+0000 7ff147fff640 10 monclient(hunting): my global_id is 4509 2024-03-25T18:22:44.164 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.162+0000 7ff147fff640 1 -- 172.21.15.97:0/642759643 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff14c004eb0 con 0x7ff160142410 2024-03-25T18:22:44.166 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.165+0000 7ff147fff640 10 client.?.objecter ms_handle_connect 0x7ff160061890 2024-03-25T18:22:44.166 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.165+0000 7ff147fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:44.167 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.165+0000 7ff147fff640 1 -- 172.21.15.97:0/642759643 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1734455139 0 0) 0x7ff160135a80 con 0x7ff1600ae280 2024-03-25T18:22:44.167 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.165+0000 7ff147fff640 10 monclient(hunting): _negotiate 2024-03-25T18:22:44.167 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.165+0000 7ff147fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:44.167 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.165+0000 7ff147fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:44.167 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.165+0000 7ff147fff640 10 monclient(hunting): my global_id is 4516 2024-03-25T18:22:44.167 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.165+0000 7ff147fff640 1 -- 172.21.15.97:0/642759643 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1601dbe00 con 0x7ff1600ae280 2024-03-25T18:22:44.178 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.176+0000 7ff147fff640 1 -- 172.21.15.97:0/642759643 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2327977566 0 0) 0x7ff1601dbe00 con 0x7ff1600ae280 2024-03-25T18:22:44.178 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.176+0000 7ff147fff640 1 -- 172.21.15.97:0/642759643 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff160135a80 con 0x7ff1600ae280 2024-03-25T18:22:44.178 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.176+0000 7ff147fff640 1 -- 172.21.15.97:0/642759643 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7ff1540019f0 con 0x7ff1600ae280 2024-03-25T18:22:44.178 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.176+0000 7ff147fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:44.178 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.176+0000 7ff147fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:22:44.178 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.176+0000 7ff147fff640 10 monclient(hunting): dump: 2024-03-25T18:22:44.178 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:22:44.178 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:22:44.178 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:22:44.179 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:22:44.179 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:44.179 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:22:44.179 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:22:44.179 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:22:44.179 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:22:44.179 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:22:44.179 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.176+0000 7ff147fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:22:44.183 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.182+0000 7ff147fff640 1 -- 172.21.15.97:0/642759643 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3008780334 0 0) 0x7ff160135d60 con 0x7ff160061890 2024-03-25T18:22:44.183 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.182+0000 7ff147fff640 10 monclient(hunting): _negotiate 2024-03-25T18:22:44.183 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.182+0000 7ff147fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:44.183 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.182+0000 7ff147fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:44.183 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.182+0000 7ff147fff640 10 monclient(hunting): my global_id is 4508 2024-03-25T18:22:44.184 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.182+0000 7ff147fff640 1 -- 172.21.15.97:0/642759643 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1601dbe00 con 0x7ff160061890 2024-03-25T18:22:44.188 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.187+0000 7ff147fff640 1 -- 172.21.15.97:0/642759643 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3601020553 0 0) 0x7ff14c004eb0 con 0x7ff160142410 2024-03-25T18:22:44.188 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.187+0000 7ff147fff640 1 -- 172.21.15.97:0/642759643 --> v1:172.21.15.97:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff160135d60 con 0x7ff160142410 2024-03-25T18:22:44.189 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.187+0000 7ff147fff640 1 -- 172.21.15.97:0/642759643 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7ff158001b00 con 0x7ff160142410 2024-03-25T18:22:44.189 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.187+0000 7ff147fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:44.189 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.187+0000 7ff147fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:22:44.189 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.187+0000 7ff147fff640 10 monclient(hunting): dump: 2024-03-25T18:22:44.189 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:22:44.189 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:22:44.189 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:22:44.189 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:22:44.189 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:44.189 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:22:44.189 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:22:44.189 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:22:44.189 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:22:44.189 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:22:44.196 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.194+0000 7ff147fff640 1 -- 172.21.15.97:0/642759643 <== mon.1 v1:172.21.15.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 760438186 0 0) 0x7ff160135a80 con 0x7ff1600ae280 2024-03-25T18:22:44.196 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.194+0000 7ff147fff640 1 -- 172.21.15.97:0/642759643 >> v1:172.21.15.97:6790/0 conn(0x7ff160061890 legacy=0x7ff160141260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:44.196 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.194+0000 7ff147fff640 1 -- 172.21.15.97:0/642759643 >> v1:172.21.15.97:6789/0 conn(0x7ff160142410 legacy=0x7ff1601db6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:44.196 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.194+0000 7ff147fff640 10 monclient: _finish_hunting 0 2024-03-25T18:22:44.196 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.194+0000 7ff147fff640 1 monclient: found mon.b 2024-03-25T18:22:44.196 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.194+0000 7ff147fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:22:44.196 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.194+0000 7ff147fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:22:44.196 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.194+0000 7ff147fff640 1 -- 172.21.15.97:0/642759643 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff160134820 con 0x7ff1600ae280 2024-03-25T18:22:44.197 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.194+0000 7ff147fff640 10 monclient: _finish_auth 0 2024-03-25T18:22:44.197 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.194+0000 7ff147fff640 10 monclient: _check_auth_tickets 2024-03-25T18:22:44.197 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.194+0000 7ff147fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:22:44.197 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.194+0000 7ff165c86640 5 monclient: authenticate success, global_id 4516 2024-03-25T18:22:44.197 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.194+0000 7ff165c86640 10 monclient: _renew_subs 2024-03-25T18:22:44.197 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.194+0000 7ff165c86640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:22:44.197 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.194+0000 7ff165c86640 1 -- 172.21.15.97:0/642759643 --> v1:172.21.15.98:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff160143020 con 0x7ff1600ae280 2024-03-25T18:22:44.197 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.194+0000 7ff165c86640 10 client.4516.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:22:44.197 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.194+0000 7ff165c86640 10 monclient: _renew_subs 2024-03-25T18:22:44.197 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.194+0000 7ff165c86640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:22:44.197 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.194+0000 7ff165c86640 1 -- 172.21.15.97:0/642759643 --> v1:172.21.15.98:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1601361f0 con 0x7ff1600ae280 2024-03-25T18:22:44.197 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.194+0000 7ff165c86640 1 librados: init done 2024-03-25T18:22:44.198 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.195+0000 7ff165c86640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:44.198 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.195+0000 7ff165c86640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:44.198 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.195+0000 7ff165c86640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:22:44.198 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.195+0000 7ff165c86640 1 -- 172.21.15.97:0/642759643 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1601410b0 con 0x7ff1600ae280 2024-03-25T18:22:44.488 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.486+0000 7ff147fff640 1 -- 172.21.15.97:0/642759643 <== mon.1 v1:172.21.15.98:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff154003610 con 0x7ff1600ae280 2024-03-25T18:22:44.488 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.486+0000 7ff147fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:22:44.488 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.486+0000 7ff147fff640 1 -- 172.21.15.97:0/642759643 <== mon.1 v1:172.21.15.98:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7ff1540039a0 con 0x7ff1600ae280 2024-03-25T18:22:44.488 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.486+0000 7ff147fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:44.488 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.486+0000 7ff147fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:22:44.489 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.486+0000 7ff147fff640 10 monclient: dump: 2024-03-25T18:22:44.489 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:22:44.489 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:22:44.489 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:22:44.489 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:22:44.489 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:44.489 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:22:44.489 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:22:44.489 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:22:44.489 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:22:44.489 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:22:44.489 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.486+0000 7ff15dffb640 4 set_mon_vals no callback set 2024-03-25T18:22:44.490 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.487+0000 7ff147fff640 1 -- 172.21.15.97:0/642759643 <== mon.1 v1:172.21.15.98:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2377622137 0 0) 0x7ff154003bc0 con 0x7ff1600ae280 2024-03-25T18:22:44.490 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.488+0000 7ff15f7fe640 10 monclient: get_auth_request con 0x7ff14c06dfb0 auth_method 0 2024-03-25T18:22:44.490 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.488+0000 7ff147fff640 10 client.4516.objecter ms_handle_connect 0x7ff14c06dfb0 2024-03-25T18:22:44.493 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.492+0000 7ff147fff640 1 -- 172.21.15.97:0/642759643 <== mon.1 v1:172.21.15.98:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 430711918 0 0) 0x7ff154054840 con 0x7ff1600ae280 2024-03-25T18:22:44.493 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.492+0000 7ff147fff640 10 client.4516.objecter ms_dispatch 0x7ff160000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:22:44.493 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.492+0000 7ff147fff640 3 client.4516.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:22:44.493 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.492+0000 7ff147fff640 3 client.4516.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:22:44.493 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.492+0000 7ff147fff640 20 client.4516.objecter dump_active .. 0 homeless 2024-03-25T18:22:44.494 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.492+0000 7ff147fff640 1 -- 172.21.15.97:0/642759643 <== mon.1 v1:172.21.15.98:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff154050350 con 0x7ff1600ae280 2024-03-25T18:22:44.494 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.492+0000 7ff147fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:44.494 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.492+0000 7ff147fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:22:44.739 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.737+0000 7ff165c86640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:44.739 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.737+0000 7ff165c86640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:44.739 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.737+0000 7ff165c86640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:22:44.739 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.737+0000 7ff165c86640 1 -- 172.21.15.97:0/642759643 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff160134d40 con 0x7ff1600ae280 2024-03-25T18:22:44.781 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.780+0000 7ff147fff640 1 -- 172.21.15.97:0/642759643 <== mon.1 v1:172.21.15.98:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2169427412) 0x7ff154052280 con 0x7ff1600ae280 2024-03-25T18:22:44.781 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.780+0000 7ff147fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:44.781 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.780+0000 7ff147fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:22:44.781 INFO:teuthology.misc.health.smithi097.stdout: 2024-03-25T18:22:44.782 INFO:teuthology.misc.health.smithi097.stdout:{"epoch":10,"fsid":"4c3d472c-79b8-486b-9e3d-2738d4306754","created":"2024-03-25T18:16:14.737309+0000","modified":"2024-03-25T18:16:20.357492+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:20.357492+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cb5152b9-262f-45c3-94b8-18d0e81df558","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6daf8793-0838-473d-a7d6-9a0d2894c75d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"136f1493-3b9c-4f43-a2e7-641482d69dae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"397d538c-4f64-4daf-9e8b-662d09f91327","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4f1df08-663f-4bda-93f8-3878ab7c24b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c4b18d70-a3ce-4b15-9b70-eae5525bd761","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8dd4033a-de0e-4f8a-96dc-59f3c1f825b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0077722e-ffa6-4a3d-9891-929d9869b286","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:22:44.784 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.782+0000 7ff165c86640 10 librados: watch_flush enter 2024-03-25T18:22:44.784 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.782+0000 7ff165c86640 10 librados: watch_flush exit 2024-03-25T18:22:44.785 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.782+0000 7ff165c86640 20 client.4516.objecter shutdown clearing up homeless session... 2024-03-25T18:22:44.785 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.782+0000 7ff165c86640 10 client.4516.objecter successfully canceled tick 2024-03-25T18:22:44.785 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.782+0000 7ff165c86640 1 -- 172.21.15.97:0/642759643 >> v1:172.21.15.98:6800/800387573 conn(0x7ff14c06dfb0 legacy=0x7ff14c070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:44.785 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.782+0000 7ff165c86640 10 monclient: shutdown 2024-03-25T18:22:44.785 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.782+0000 7ff165c86640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:22:44.785 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.782+0000 7ff165c86640 1 -- 172.21.15.97:0/642759643 >> v1:172.21.15.98:6789/0 conn(0x7ff1600ae280 legacy=0x7ff1601dafe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:44.785 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.783+0000 7ff165c86640 1 -- 172.21.15.97:0/642759643 shutdown_connections 2024-03-25T18:22:44.785 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.783+0000 7ff165c86640 1 -- 172.21.15.97:0/642759643 >> 172.21.15.97:0/642759643 conn(0x7ff1600b0ef0 msgr2=0x7ff1600b0510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:22:44.785 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.783+0000 7ff165c86640 1 -- 172.21.15.97:0/642759643 shutdown_connections 2024-03-25T18:22:44.785 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.783+0000 7ff165c86640 1 -- 172.21.15.97:0/642759643 wait complete. 2024-03-25T18:22:44.785 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:44.783+0000 7ff165c86640 1 librados: shutdown 2024-03-25T18:22:44.798 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:22:51.604 DEBUG:teuthology.orchestra.run.smithi097:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:22:51.725 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:51.723+0000 7f10a5088640 10 monclient: get_monmap_and_config 2024-03-25T18:22:51.725 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:51.723+0000 7f10a5088640 10 monclient: build_initial_monmap 2024-03-25T18:22:51.725 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:51.723+0000 7f10a5088640 10 monclient: monmap: 2024-03-25T18:22:51.725 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:22:51.725 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:22:51.725 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:22:51.725037+0000 2024-03-25T18:22:51.725 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:22:51.725037+0000 2024-03-25T18:22:51.725 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:22:51.725 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:22:51.726 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:22:51.726 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T18:22:51.726 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-b 2024-03-25T18:22:51.726 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:22:51.727 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:51.724+0000 7f10a5088640 1 Processor -- start 2024-03-25T18:22:51.727 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:51.724+0000 7f10a5088640 1 -- start start 2024-03-25T18:22:51.727 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:51.724+0000 7f10a5088640 10 monclient: init 2024-03-25T18:22:51.727 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:51.725+0000 7f10a5088640 10 monclient: _reopen_session rank -1 2024-03-25T18:22:51.727 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:51.725+0000 7f10a5088640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:22:51.727 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:51.725+0000 7f10a5088640 10 monclient(hunting): picked mon.noname-a con 0x7f10a00a98f0 addr v1:172.21.15.97:6789/0 2024-03-25T18:22:51.727 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:51.725+0000 7f10a5088640 10 monclient(hunting): picked mon.noname-b con 0x7f10a013f710 addr v1:172.21.15.97:6790/0 2024-03-25T18:22:51.727 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:51.725+0000 7f10a5088640 10 monclient(hunting): picked mon.noname-c con 0x7f10a01430c0 addr v1:172.21.15.98:6789/0 2024-03-25T18:22:51.728 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:51.725+0000 7f10a5088640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10a0142520 con 0x7f10a00a98f0 2024-03-25T18:22:51.728 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:51.725+0000 7f10a5088640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10a00b28a0 con 0x7f10a01430c0 2024-03-25T18:22:51.728 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:51.725+0000 7f10a5088640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10a00b2a30 con 0x7f10a013f710 2024-03-25T18:22:51.728 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:51.725+0000 7f10a5088640 10 monclient(hunting): _renew_subs 2024-03-25T18:22:51.728 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:51.725+0000 7f10a5088640 10 monclient(hunting): authenticate will time out at 7984.001383s 2024-03-25T18:22:51.736 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:51.735+0000 7f109ed9d640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f10a00a98f0 0x7f10a00a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:53044/0 (socket says 172.21.15.97:53044) 2024-03-25T18:22:51.736 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:51.735+0000 7f109ed9d640 1 -- 172.21.15.97:0/3694057903 learned_addr learned my addr 172.21.15.97:0/3694057903 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:22:51.756 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:51.754+0000 7f109dd9b640 1 -- 172.21.15.97:0/3694057903 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 452532433 0 0) 0x7f10a0142520 con 0x7f10a00a98f0 2024-03-25T18:22:51.757 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:51.754+0000 7f109dd9b640 10 monclient(hunting): _negotiate 2024-03-25T18:22:51.757 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:51.754+0000 7f109dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:51.757 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:51.754+0000 7f109dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:51.757 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:51.754+0000 7f109dd9b640 10 monclient(hunting): my global_id is 4512 2024-03-25T18:22:51.757 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:51.754+0000 7f109dd9b640 1 -- 172.21.15.97:0/3694057903 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1088003520 con 0x7f10a00a98f0 2024-03-25T18:22:51.762 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:51.761+0000 7f109dd9b640 1 -- 172.21.15.97:0/3694057903 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2415440827 0 0) 0x7f10a00b2a30 con 0x7f10a013f710 2024-03-25T18:22:51.762 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:51.761+0000 7f109dd9b640 10 monclient(hunting): _negotiate 2024-03-25T18:22:51.762 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:51.761+0000 7f109dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:51.763 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:51.761+0000 7f109dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:51.763 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:51.761+0000 7f109dd9b640 10 monclient(hunting): my global_id is 4511 2024-03-25T18:22:51.763 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:51.761+0000 7f109dd9b640 1 -- 172.21.15.97:0/3694057903 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10a0142520 con 0x7f10a013f710 2024-03-25T18:22:51.772 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:51.771+0000 7f109dd9b640 1 -- 172.21.15.97:0/3694057903 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3370636603 0 0) 0x7f10a00b28a0 con 0x7f10a01430c0 2024-03-25T18:22:51.773 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:51.771+0000 7f109dd9b640 10 monclient(hunting): _negotiate 2024-03-25T18:22:51.773 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:51.771+0000 7f109dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:51.773 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:51.771+0000 7f109dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:51.773 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:51.771+0000 7f109dd9b640 10 monclient(hunting): my global_id is 4519 2024-03-25T18:22:51.773 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:51.771+0000 7f109dd9b640 1 -- 172.21.15.97:0/3694057903 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10a00b2a30 con 0x7f10a01430c0 2024-03-25T18:22:51.782 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:51.780+0000 7f109dd9b640 1 -- 172.21.15.97:0/3694057903 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 722294092 0 0) 0x7f1088003520 con 0x7f10a00a98f0 2024-03-25T18:22:51.782 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:51.780+0000 7f109dd9b640 1 -- 172.21.15.97:0/3694057903 >> v1:172.21.15.97:6790/0 conn(0x7f10a013f710 legacy=0x7f10a01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:51.782 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:51.780+0000 7f109dd9b640 1 -- 172.21.15.97:0/3694057903 >> v1:172.21.15.98:6789/0 conn(0x7f10a01430c0 legacy=0x7f10a014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:51.782 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:51.780+0000 7f109dd9b640 10 monclient: _finish_hunting 0 2024-03-25T18:22:51.782 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:51.780+0000 7f109dd9b640 1 monclient: found mon.noname-a 2024-03-25T18:22:51.782 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:51.780+0000 7f109dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:22:51.782 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:51.780+0000 7f109dd9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.97:6789/0 2024-03-25T18:22:51.782 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:51.780+0000 7f109dd9b640 1 -- 172.21.15.97:0/3694057903 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10a00b44c0 con 0x7f10a00a98f0 2024-03-25T18:22:51.783 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:51.780+0000 7f109dd9b640 10 monclient: _finish_auth 0 2024-03-25T18:22:51.783 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:51.780+0000 7f109dd9b640 10 monclient: _check_auth_tickets 2024-03-25T18:22:51.783 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:51.780+0000 7f109dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:22:51.783 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:51.780+0000 7f109dd9b640 1 -- 172.21.15.97:0/3694057903 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f1090001690 con 0x7f10a00a98f0 2024-03-25T18:22:51.783 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:51.780+0000 7f109dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:51.783 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:51.780+0000 7f109dd9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:22:51.783 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:51.780+0000 7f109dd9b640 10 monclient: dump: 2024-03-25T18:22:51.783 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:22:51.783 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:22:51.783 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:22:51.783 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:22:51.783 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:51.783 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:22:51.783 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:22:51.783 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:22:51.784 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:22:51.784 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:22:51.784 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:51.780+0000 7f10a5088640 5 monclient: authenticate success, global_id 4512 2024-03-25T18:22:51.784 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:51.780+0000 7f10a5088640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:22:51.798 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:51.796+0000 7f109dd9b640 1 -- 172.21.15.97:0/3694057903 <== mon.0 v1:172.21.15.97:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f10900017f0 con 0x7f10a00a98f0 2024-03-25T18:22:51.798 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:51.796+0000 7f109dd9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:22:51.798 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:51.796+0000 7f109dd9b640 1 -- 172.21.15.97:0/3694057903 <== mon.0 v1:172.21.15.97:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f1090001c00 con 0x7f10a00a98f0 2024-03-25T18:22:51.798 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:51.796+0000 7f10a5088640 10 monclient: get_monmap_and_config success 2024-03-25T18:22:51.798 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:51.796+0000 7f10a5088640 4 set_mon_vals no callback set 2024-03-25T18:22:51.798 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:51.796+0000 7f109dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:51.798 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:51.796+0000 7f109dd9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:22:51.798 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:51.796+0000 7f109dd9b640 10 monclient: dump: 2024-03-25T18:22:51.798 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:22:51.798 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:22:51.798 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:22:51.798 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:22:51.798 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:51.799 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:22:51.799 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:22:51.799 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:22:51.799 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:22:51.799 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:22:51.801 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:51.797+0000 7f10a5088640 10 monclient: shutdown 2024-03-25T18:22:51.801 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:51.797+0000 7f10a5088640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:22:51.801 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:51.797+0000 7f10a5088640 1 -- 172.21.15.97:0/3694057903 >> v1:172.21.15.97:6789/0 conn(0x7f10a00a98f0 legacy=0x7f10a00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:51.801 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:51.797+0000 7f10a5088640 1 -- 172.21.15.97:0/3694057903 shutdown_connections 2024-03-25T18:22:51.801 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:51.797+0000 7f10a5088640 1 -- 172.21.15.97:0/3694057903 >> 172.21.15.97:0/3694057903 conn(0x7f10a00b7ea0 msgr2=0x7f10a00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:22:51.801 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:51.797+0000 7f10a5088640 1 -- 172.21.15.97:0/3694057903 shutdown_connections 2024-03-25T18:22:51.801 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:51.797+0000 7f10a5088640 1 -- 172.21.15.97:0/3694057903 wait complete. 2024-03-25T18:22:51.801 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:51.798+0000 7f10a5088640 10 monclient: build_initial_monmap 2024-03-25T18:22:51.801 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:51.798+0000 7f10a5088640 10 monclient: monmap: 2024-03-25T18:22:51.802 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:22:51.802 INFO:teuthology.misc.health.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:22:51.802 INFO:teuthology.misc.health.smithi097.stderr:last_changed 0.000000 2024-03-25T18:22:51.802 INFO:teuthology.misc.health.smithi097.stderr:created 0.000000 2024-03-25T18:22:51.802 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:22:51.802 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 1 2024-03-25T18:22:51.802 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:22:51.802 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T18:22:51.802 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-c 2024-03-25T18:22:51.802 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:22:51.802 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:51.798+0000 7f10a5088640 1 librados: starting msgr at 2024-03-25T18:22:51.802 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:51.798+0000 7f10a5088640 1 librados: starting objecter 2024-03-25T18:22:51.802 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:51.798+0000 7f10a5088640 1 Processor -- start 2024-03-25T18:22:51.803 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:51.799+0000 7f10a5088640 1 -- start start 2024-03-25T18:22:51.803 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:51.799+0000 7f10a5088640 1 librados: setting wanted keys 2024-03-25T18:22:51.803 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:51.799+0000 7f10a5088640 1 librados: calling monclient init 2024-03-25T18:22:51.803 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:51.799+0000 7f10a5088640 10 monclient: init 2024-03-25T18:22:51.803 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:51.799+0000 7f10a5088640 10 monclient: _reopen_session rank -1 2024-03-25T18:22:51.803 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:51.799+0000 7f10a5088640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:22:51.803 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:51.799+0000 7f10a5088640 10 monclient(hunting): picked mon.noname-c con 0x7f10a00a98f0 addr v1:172.21.15.97:6790/0 2024-03-25T18:22:51.803 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:51.799+0000 7f10a5088640 10 monclient(hunting): picked mon.noname-a con 0x7f10a013f710 addr v1:172.21.15.97:6789/0 2024-03-25T18:22:51.803 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:51.799+0000 7f10a5088640 10 monclient(hunting): picked mon.noname-b con 0x7f10a01430c0 addr v1:172.21.15.98:6789/0 2024-03-25T18:22:51.803 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:51.799+0000 7f10a5088640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10a0138c70 con 0x7f10a013f710 2024-03-25T18:22:51.803 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:51.799+0000 7f10a5088640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10a01367d0 con 0x7f10a01430c0 2024-03-25T18:22:51.803 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:51.799+0000 7f10a5088640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10a0136960 con 0x7f10a00a98f0 2024-03-25T18:22:51.803 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:51.799+0000 7f10a5088640 10 monclient(hunting): _renew_subs 2024-03-25T18:22:51.804 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:51.799+0000 7f10a5088640 10 monclient(hunting): authenticate will time out at 7984.075514s 2024-03-25T18:22:51.813 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:51.812+0000 7f109ed9d640 1 --1- >> v1:172.21.15.97:6790/0 conn(0x7f10a00a98f0 0x7f10a0137610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.97:6790/0 says I am v1:172.21.15.97:38160/0 (socket says 172.21.15.97:38160) 2024-03-25T18:22:51.814 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:51.812+0000 7f109ed9d640 1 -- 172.21.15.97:0/1581853280 learned_addr learned my addr 172.21.15.97:0/1581853280 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:22:51.818 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:51.817+0000 7f10877fe640 10 client.?.objecter ms_handle_connect 0x7f10a00a98f0 2024-03-25T18:22:51.818 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:51.817+0000 7f10877fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:51.821 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:51.820+0000 7f10877fe640 10 client.?.objecter ms_handle_connect 0x7f10a013f710 2024-03-25T18:22:51.822 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:51.820+0000 7f10877fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:51.823 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:51.821+0000 7f10877fe640 1 -- 172.21.15.97:0/1581853280 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 261519624 0 0) 0x7f10a0136960 con 0x7f10a00a98f0 2024-03-25T18:22:51.823 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:51.821+0000 7f10877fe640 10 monclient(hunting): _negotiate 2024-03-25T18:22:51.823 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:51.821+0000 7f10877fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:51.823 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:51.821+0000 7f10877fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:51.823 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:51.821+0000 7f10877fe640 10 monclient(hunting): my global_id is 4514 2024-03-25T18:22:51.823 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:51.821+0000 7f10877fe640 1 -- 172.21.15.97:0/1581853280 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f108c004eb0 con 0x7f10a00a98f0 2024-03-25T18:22:51.832 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:51.831+0000 7f10877fe640 10 client.?.objecter ms_handle_connect 0x7f10a01430c0 2024-03-25T18:22:51.832 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:51.831+0000 7f10877fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:51.841 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:51.839+0000 7f10877fe640 1 -- 172.21.15.97:0/1581853280 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3755478044 0 0) 0x7f10a0138c70 con 0x7f10a013f710 2024-03-25T18:22:51.841 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:51.839+0000 7f10877fe640 10 monclient(hunting): _negotiate 2024-03-25T18:22:51.841 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:51.839+0000 7f10877fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:51.841 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:51.839+0000 7f10877fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:51.841 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:51.839+0000 7f10877fe640 10 monclient(hunting): my global_id is 4515 2024-03-25T18:22:51.841 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:51.839+0000 7f10877fe640 1 -- 172.21.15.97:0/1581853280 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10a0136960 con 0x7f10a013f710 2024-03-25T18:22:51.842 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:51.840+0000 7f10877fe640 1 -- 172.21.15.97:0/1581853280 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3101716534 0 0) 0x7f10a01367d0 con 0x7f10a01430c0 2024-03-25T18:22:51.842 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:51.840+0000 7f10877fe640 10 monclient(hunting): _negotiate 2024-03-25T18:22:51.842 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:51.840+0000 7f10877fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:51.842 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:51.840+0000 7f10877fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:51.842 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:51.840+0000 7f10877fe640 10 monclient(hunting): my global_id is 4522 2024-03-25T18:22:51.842 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:51.840+0000 7f10877fe640 1 -- 172.21.15.97:0/1581853280 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10a0138c70 con 0x7f10a01430c0 2024-03-25T18:22:51.844 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:51.842+0000 7f10877fe640 1 -- 172.21.15.97:0/1581853280 <== mon.2 v1:172.21.15.97:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3038690127 0 0) 0x7f108c004eb0 con 0x7f10a00a98f0 2024-03-25T18:22:51.844 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:51.842+0000 7f10877fe640 1 -- 172.21.15.97:0/1581853280 --> v1:172.21.15.97:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f10a01367d0 con 0x7f10a00a98f0 2024-03-25T18:22:51.844 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:51.842+0000 7f10877fe640 1 -- 172.21.15.97:0/1581853280 <== mon.2 v1:172.21.15.97:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f1090001bc0 con 0x7f10a00a98f0 2024-03-25T18:22:51.844 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:51.842+0000 7f10877fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:51.844 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:51.842+0000 7f10877fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:22:51.844 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:51.842+0000 7f10877fe640 10 monclient(hunting): dump: 2024-03-25T18:22:51.844 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:22:51.844 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:22:51.844 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:22:51.844 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:22:51.845 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:51.845 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:22:51.845 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:22:51.845 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:22:51.845 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:22:51.845 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:22:51.845 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:51.842+0000 7f10877fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:22:51.855 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:51.853+0000 7f10877fe640 1 -- 172.21.15.97:0/1581853280 <== mon.2 v1:172.21.15.97:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4091575154 0 0) 0x7f10a01367d0 con 0x7f10a00a98f0 2024-03-25T18:22:51.855 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:51.853+0000 7f10877fe640 1 -- 172.21.15.97:0/1581853280 >> v1:172.21.15.98:6789/0 conn(0x7f10a01430c0 legacy=0x7f10a01e2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:51.855 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:51.853+0000 7f10877fe640 1 -- 172.21.15.97:0/1581853280 >> v1:172.21.15.97:6789/0 conn(0x7f10a013f710 legacy=0x7f10a0132f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:51.855 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:51.853+0000 7f10877fe640 10 monclient: _finish_hunting 0 2024-03-25T18:22:51.856 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:51.853+0000 7f10877fe640 1 monclient: found mon.c 2024-03-25T18:22:51.856 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:51.853+0000 7f10877fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:22:51.856 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:51.853+0000 7f10877fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.97:6790/0 2024-03-25T18:22:51.856 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:51.853+0000 7f10877fe640 1 -- 172.21.15.97:0/1581853280 --> v1:172.21.15.97:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10a0139630 con 0x7f10a00a98f0 2024-03-25T18:22:51.856 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:51.853+0000 7f10877fe640 10 monclient: _finish_auth 0 2024-03-25T18:22:51.856 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:51.853+0000 7f10877fe640 10 monclient: _check_auth_tickets 2024-03-25T18:22:51.856 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:51.853+0000 7f10877fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:22:51.856 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:51.853+0000 7f10a5088640 5 monclient: authenticate success, global_id 4514 2024-03-25T18:22:51.856 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:51.853+0000 7f10a5088640 10 monclient: _renew_subs 2024-03-25T18:22:51.856 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:51.853+0000 7f10a5088640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.97:6790/0 2024-03-25T18:22:51.856 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:51.853+0000 7f10a5088640 1 -- 172.21.15.97:0/1581853280 --> v1:172.21.15.97:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10a0139880 con 0x7f10a00a98f0 2024-03-25T18:22:51.856 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:51.853+0000 7f10a5088640 10 client.4514.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:22:51.856 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:51.853+0000 7f10a5088640 10 monclient: _renew_subs 2024-03-25T18:22:51.856 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:51.853+0000 7f10a5088640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.97:6790/0 2024-03-25T18:22:51.857 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:51.853+0000 7f10a5088640 1 -- 172.21.15.97:0/1581853280 --> v1:172.21.15.97:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f10a01337f0 con 0x7f10a00a98f0 2024-03-25T18:22:51.857 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:51.853+0000 7f10a5088640 1 librados: init done 2024-03-25T18:22:51.857 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:51.854+0000 7f10a5088640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:51.857 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:51.854+0000 7f10a5088640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:51.857 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:51.854+0000 7f10a5088640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.97:6790/0 2024-03-25T18:22:51.857 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:51.854+0000 7f10a5088640 1 -- 172.21.15.97:0/1581853280 --> v1:172.21.15.97:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10a00627b0 con 0x7f10a00a98f0 2024-03-25T18:22:52.153 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:52.151+0000 7f10877fe640 1 -- 172.21.15.97:0/1581853280 <== mon.2 v1:172.21.15.97:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f10900011f0 con 0x7f10a00a98f0 2024-03-25T18:22:52.153 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:52.151+0000 7f10877fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:22:52.153 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:52.151+0000 7f10877fe640 1 -- 172.21.15.97:0/1581853280 <== mon.2 v1:172.21.15.97:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f1090004cb0 con 0x7f10a00a98f0 2024-03-25T18:22:52.153 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:52.151+0000 7f10877fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:52.153 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:52.151+0000 7f10877fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:22:52.153 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:52.151+0000 7f10877fe640 10 monclient: dump: 2024-03-25T18:22:52.153 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:22:52.154 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:22:52.154 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:22:52.154 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:22:52.154 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:52.154 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:22:52.154 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:22:52.154 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:22:52.154 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:22:52.154 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:22:52.154 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:52.151+0000 7f109d59a640 4 set_mon_vals no callback set 2024-03-25T18:22:52.155 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:52.152+0000 7f10877fe640 1 -- 172.21.15.97:0/1581853280 <== mon.2 v1:172.21.15.97:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2377622137 0 0) 0x7f109001ad00 con 0x7f10a00a98f0 2024-03-25T18:22:52.155 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:52.153+0000 7f109e59c640 10 monclient: get_auth_request con 0x7f108c06dd50 auth_method 0 2024-03-25T18:22:52.155 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:52.153+0000 7f10877fe640 10 client.4514.objecter ms_handle_connect 0x7f108c06dd50 2024-03-25T18:22:52.159 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:52.157+0000 7f10877fe640 1 -- 172.21.15.97:0/1581853280 <== mon.2 v1:172.21.15.97:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 430711918 0 0) 0x7f10900549d0 con 0x7f10a00a98f0 2024-03-25T18:22:52.159 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:52.157+0000 7f10877fe640 10 client.4514.objecter ms_dispatch 0x7f10a0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:22:52.159 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:52.157+0000 7f10877fe640 3 client.4514.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:22:52.159 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:52.157+0000 7f10877fe640 3 client.4514.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:22:52.159 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:52.157+0000 7f10877fe640 20 client.4514.objecter dump_active .. 0 homeless 2024-03-25T18:22:52.159 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:52.157+0000 7f10877fe640 1 -- 172.21.15.97:0/1581853280 <== mon.2 v1:172.21.15.97:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1090050310 con 0x7f10a00a98f0 2024-03-25T18:22:52.159 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:52.158+0000 7f10877fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:52.159 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:52.158+0000 7f10877fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:22:52.420 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:52.419+0000 7f10a5088640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:52.420 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:52.419+0000 7f10a5088640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:52.420 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:52.419+0000 7f10a5088640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.97:6790/0 2024-03-25T18:22:52.421 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:52.419+0000 7f10a5088640 1 -- 172.21.15.97:0/1581853280 --> v1:172.21.15.97:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f10a0069f40 con 0x7f10a00a98f0 2024-03-25T18:22:52.536 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:52.535+0000 7f10877fe640 1 -- 172.21.15.97:0/1581853280 <== mon.2 v1:172.21.15.97:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2169427412) 0x7f1090052410 con 0x7f10a00a98f0 2024-03-25T18:22:52.536 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:52.535+0000 7f10877fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:52.537 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:52.535+0000 7f10877fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:22:52.537 INFO:teuthology.misc.health.smithi097.stdout: 2024-03-25T18:22:52.537 INFO:teuthology.misc.health.smithi097.stdout:{"epoch":10,"fsid":"4c3d472c-79b8-486b-9e3d-2738d4306754","created":"2024-03-25T18:16:14.737309+0000","modified":"2024-03-25T18:16:20.357492+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:20.357492+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cb5152b9-262f-45c3-94b8-18d0e81df558","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6daf8793-0838-473d-a7d6-9a0d2894c75d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"136f1493-3b9c-4f43-a2e7-641482d69dae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"397d538c-4f64-4daf-9e8b-662d09f91327","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4f1df08-663f-4bda-93f8-3878ab7c24b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c4b18d70-a3ce-4b15-9b70-eae5525bd761","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8dd4033a-de0e-4f8a-96dc-59f3c1f825b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0077722e-ffa6-4a3d-9891-929d9869b286","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:22:52.539 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:52.537+0000 7f10a5088640 10 librados: watch_flush enter 2024-03-25T18:22:52.539 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:52.537+0000 7f10a5088640 10 librados: watch_flush exit 2024-03-25T18:22:52.539 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:52.537+0000 7f10a5088640 20 client.4514.objecter shutdown clearing up homeless session... 2024-03-25T18:22:52.539 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:52.537+0000 7f10a5088640 10 client.4514.objecter successfully canceled tick 2024-03-25T18:22:52.539 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:52.537+0000 7f10a5088640 1 -- 172.21.15.97:0/1581853280 >> v1:172.21.15.98:6800/800387573 conn(0x7f108c06dd50 legacy=0x7f108c070210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:52.540 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:52.537+0000 7f10a5088640 10 monclient: shutdown 2024-03-25T18:22:52.540 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:52.537+0000 7f10a5088640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:22:52.540 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:52.537+0000 7f10a5088640 1 -- 172.21.15.97:0/1581853280 >> v1:172.21.15.97:6790/0 conn(0x7f10a00a98f0 legacy=0x7f10a0137610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:52.540 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:52.537+0000 7f10a5088640 1 -- 172.21.15.97:0/1581853280 shutdown_connections 2024-03-25T18:22:52.540 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:52.537+0000 7f10a5088640 1 -- 172.21.15.97:0/1581853280 >> 172.21.15.97:0/1581853280 conn(0x7f10a00b7ea0 msgr2=0x7f10a0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:22:52.540 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:52.537+0000 7f10a5088640 1 -- 172.21.15.97:0/1581853280 shutdown_connections 2024-03-25T18:22:52.540 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:52.537+0000 7f10a5088640 1 -- 172.21.15.97:0/1581853280 wait complete. 2024-03-25T18:22:52.540 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:52.538+0000 7f10a5088640 1 librados: shutdown 2024-03-25T18:22:52.551 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:22:53.095 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:06:55.133 34931== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:53.099 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:06:55.138 34931== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:53.132 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:06:55.171 34931== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:53.147 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:06:55.185 34935== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:53.162 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:06:55.201 34935== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:53.172 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:06:55.215 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:53.177 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:06:55.220 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:53.199 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:06:55.238 34935== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:53.200 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:06:55.242 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:59.356 DEBUG:teuthology.orchestra.run.smithi097:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:22:59.470 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.468+0000 7f478cee6640 10 monclient: get_monmap_and_config 2024-03-25T18:22:59.470 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.468+0000 7f478cee6640 10 monclient: build_initial_monmap 2024-03-25T18:22:59.470 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.468+0000 7f478cee6640 10 monclient: monmap: 2024-03-25T18:22:59.470 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:22:59.470 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:22:59.470 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:22:59.469621+0000 2024-03-25T18:22:59.470 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:22:59.469621+0000 2024-03-25T18:22:59.471 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:22:59.471 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:22:59.471 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:22:59.471 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T18:22:59.471 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-b 2024-03-25T18:22:59.471 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:22:59.471 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.469+0000 7f478cee6640 1 Processor -- start 2024-03-25T18:22:59.471 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.469+0000 7f478cee6640 1 -- start start 2024-03-25T18:22:59.471 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.469+0000 7f478cee6640 10 monclient: init 2024-03-25T18:22:59.472 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.469+0000 7f478cee6640 10 monclient: _reopen_session rank -1 2024-03-25T18:22:59.472 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.469+0000 7f478cee6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:22:59.472 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.469+0000 7f478cee6640 10 monclient(hunting): picked mon.noname-b con 0x7f47880a98f0 addr v1:172.21.15.97:6790/0 2024-03-25T18:22:59.472 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.469+0000 7f478cee6640 10 monclient(hunting): picked mon.noname-a con 0x7f478813f710 addr v1:172.21.15.97:6789/0 2024-03-25T18:22:59.472 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.469+0000 7f478cee6640 10 monclient(hunting): picked mon.noname-c con 0x7f47881430c0 addr v1:172.21.15.98:6789/0 2024-03-25T18:22:59.473 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.469+0000 7f478cee6640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4788142520 con 0x7f478813f710 2024-03-25T18:22:59.473 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.469+0000 7f478cee6640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47880b28a0 con 0x7f47881430c0 2024-03-25T18:22:59.473 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.470+0000 7f478cee6640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47880b2a30 con 0x7f47880a98f0 2024-03-25T18:22:59.473 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.470+0000 7f478cee6640 10 monclient(hunting): _renew_subs 2024-03-25T18:22:59.473 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.470+0000 7f478cee6640 10 monclient(hunting): authenticate will time out at 7991.745884s 2024-03-25T18:22:59.485 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.484+0000 7f4786d9d640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f47881430c0 0x7f478814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.97:56908/0 (socket says 172.21.15.97:56908) 2024-03-25T18:22:59.485 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.484+0000 7f4786d9d640 1 -- 172.21.15.97:0/222391359 learned_addr learned my addr 172.21.15.97:0/222391359 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:22:59.494 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.492+0000 7f478559a640 1 -- 172.21.15.97:0/222391359 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2555546349 0 0) 0x7f47880b28a0 con 0x7f47881430c0 2024-03-25T18:22:59.494 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.492+0000 7f478559a640 10 monclient(hunting): _negotiate 2024-03-25T18:22:59.494 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.492+0000 7f478559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:59.494 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.492+0000 7f478559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:59.494 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.492+0000 7f478559a640 10 monclient(hunting): my global_id is 4525 2024-03-25T18:22:59.494 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.492+0000 7f478559a640 1 -- 172.21.15.97:0/222391359 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4770003520 con 0x7f47881430c0 2024-03-25T18:22:59.503 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.498+0000 7f478559a640 1 -- 172.21.15.97:0/222391359 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3743206681 0 0) 0x7f4770003520 con 0x7f47881430c0 2024-03-25T18:22:59.503 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.498+0000 7f478559a640 1 -- 172.21.15.97:0/222391359 >> v1:172.21.15.97:6790/0 conn(0x7f47880a98f0 legacy=0x7f47880a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:59.503 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.498+0000 7f478559a640 1 -- 172.21.15.97:0/222391359 >> v1:172.21.15.97:6789/0 conn(0x7f478813f710 legacy=0x7f47881429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:59.503 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.498+0000 7f478559a640 10 monclient: _finish_hunting 0 2024-03-25T18:22:59.503 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.498+0000 7f478559a640 1 monclient: found mon.noname-c 2024-03-25T18:22:59.504 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.498+0000 7f478559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:22:59.504 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.498+0000 7f478559a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.98:6789/0 2024-03-25T18:22:59.504 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.498+0000 7f478559a640 1 -- 172.21.15.97:0/222391359 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47880b44c0 con 0x7f47881430c0 2024-03-25T18:22:59.504 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.498+0000 7f478559a640 10 monclient: _finish_auth 0 2024-03-25T18:22:59.504 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.498+0000 7f478559a640 10 monclient: _check_auth_tickets 2024-03-25T18:22:59.504 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.498+0000 7f478559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:22:59.504 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.498+0000 7f478559a640 1 -- 172.21.15.97:0/222391359 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f4780001a10 con 0x7f47881430c0 2024-03-25T18:22:59.504 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.498+0000 7f478559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:59.504 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.498+0000 7f478559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:22:59.504 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.498+0000 7f478559a640 10 monclient: dump: 2024-03-25T18:22:59.504 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:22:59.505 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:22:59.505 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:22:59.505 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:22:59.505 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:59.505 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:22:59.505 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:22:59.505 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:22:59.505 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:22:59.505 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:22:59.505 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.498+0000 7f478cee6640 5 monclient: authenticate success, global_id 4525 2024-03-25T18:22:59.505 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.498+0000 7f478cee6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:22:59.513 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.512+0000 7f478559a640 1 -- 172.21.15.97:0/222391359 <== mon.1 v1:172.21.15.98:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4780001b70 con 0x7f47881430c0 2024-03-25T18:22:59.513 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.512+0000 7f478559a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:22:59.513 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.512+0000 7f478559a640 1 -- 172.21.15.97:0/222391359 <== mon.1 v1:172.21.15.98:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f4780001fc0 con 0x7f47881430c0 2024-03-25T18:22:59.514 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.512+0000 7f478cee6640 10 monclient: get_monmap_and_config success 2024-03-25T18:22:59.514 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.512+0000 7f478cee6640 4 set_mon_vals no callback set 2024-03-25T18:22:59.514 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.512+0000 7f478559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:59.514 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.512+0000 7f478559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:22:59.514 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.512+0000 7f478559a640 10 monclient: dump: 2024-03-25T18:22:59.514 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:22:59.514 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:22:59.514 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:22:59.514 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:22:59.514 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:59.514 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:22:59.515 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:22:59.515 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:22:59.515 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:22:59.515 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:22:59.517 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.513+0000 7f478cee6640 10 monclient: shutdown 2024-03-25T18:22:59.517 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.513+0000 7f478cee6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:22:59.517 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.513+0000 7f478cee6640 1 -- 172.21.15.97:0/222391359 >> v1:172.21.15.98:6789/0 conn(0x7f47881430c0 legacy=0x7f478814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:59.517 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.513+0000 7f478cee6640 1 -- 172.21.15.97:0/222391359 shutdown_connections 2024-03-25T18:22:59.517 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.513+0000 7f478cee6640 1 -- 172.21.15.97:0/222391359 >> 172.21.15.97:0/222391359 conn(0x7f47880b7ea0 msgr2=0x7f47880b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:22:59.517 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.513+0000 7f478cee6640 1 -- 172.21.15.97:0/222391359 shutdown_connections 2024-03-25T18:22:59.517 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.513+0000 7f478cee6640 1 -- 172.21.15.97:0/222391359 wait complete. 2024-03-25T18:22:59.517 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.514+0000 7f478cee6640 10 monclient: build_initial_monmap 2024-03-25T18:22:59.517 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.514+0000 7f478cee6640 10 monclient: monmap: 2024-03-25T18:22:59.517 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:22:59.517 INFO:teuthology.misc.health.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:22:59.517 INFO:teuthology.misc.health.smithi097.stderr:last_changed 0.000000 2024-03-25T18:22:59.518 INFO:teuthology.misc.health.smithi097.stderr:created 0.000000 2024-03-25T18:22:59.518 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:22:59.518 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 1 2024-03-25T18:22:59.518 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:22:59.518 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T18:22:59.518 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-c 2024-03-25T18:22:59.518 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:22:59.518 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.514+0000 7f478cee6640 1 librados: starting msgr at 2024-03-25T18:22:59.518 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.514+0000 7f478cee6640 1 librados: starting objecter 2024-03-25T18:22:59.518 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.514+0000 7f478cee6640 1 Processor -- start 2024-03-25T18:22:59.518 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.514+0000 7f478cee6640 1 -- start start 2024-03-25T18:22:59.518 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.514+0000 7f478cee6640 1 librados: setting wanted keys 2024-03-25T18:22:59.519 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.514+0000 7f478cee6640 1 librados: calling monclient init 2024-03-25T18:22:59.519 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.514+0000 7f478cee6640 10 monclient: init 2024-03-25T18:22:59.519 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.515+0000 7f478cee6640 10 monclient: _reopen_session rank -1 2024-03-25T18:22:59.519 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.515+0000 7f478cee6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:22:59.519 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.515+0000 7f478cee6640 10 monclient(hunting): picked mon.noname-c con 0x7f47880a98f0 addr v1:172.21.15.97:6790/0 2024-03-25T18:22:59.519 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.515+0000 7f478cee6640 10 monclient(hunting): picked mon.noname-a con 0x7f478813f710 addr v1:172.21.15.97:6789/0 2024-03-25T18:22:59.519 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.515+0000 7f478cee6640 10 monclient(hunting): picked mon.noname-b con 0x7f47881430c0 addr v1:172.21.15.98:6789/0 2024-03-25T18:22:59.519 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.515+0000 7f478cee6640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4788136900 con 0x7f478813f710 2024-03-25T18:22:59.519 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.515+0000 7f478cee6640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4788136d90 con 0x7f47881430c0 2024-03-25T18:22:59.519 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.515+0000 7f478cee6640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47881397b0 con 0x7f47880a98f0 2024-03-25T18:22:59.519 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.515+0000 7f478cee6640 10 monclient(hunting): _renew_subs 2024-03-25T18:22:59.519 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.515+0000 7f478cee6640 10 monclient(hunting): authenticate will time out at 7991.791418s 2024-03-25T18:22:59.525 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.524+0000 7f4786d9d640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f47881430c0 0x7f47881e1e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.97:56974/0 (socket says 172.21.15.97:56974) 2024-03-25T18:22:59.525 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.524+0000 7f4786d9d640 1 -- 172.21.15.97:0/4207727195 learned_addr learned my addr 172.21.15.97:0/4207727195 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:22:59.526 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.525+0000 7f4776ffd640 10 client.?.objecter ms_handle_connect 0x7f47881430c0 2024-03-25T18:22:59.526 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.525+0000 7f4776ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:59.539 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.537+0000 7f4776ffd640 10 client.?.objecter ms_handle_connect 0x7f478813f710 2024-03-25T18:22:59.539 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.537+0000 7f4776ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:59.541 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.540+0000 7f4776ffd640 1 -- 172.21.15.97:0/4207727195 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1572775394 0 0) 0x7f4788136d90 con 0x7f47881430c0 2024-03-25T18:22:59.541 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.540+0000 7f4776ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:22:59.542 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.540+0000 7f4776ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:59.542 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.540+0000 7f4776ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:59.542 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.540+0000 7f4776ffd640 10 monclient(hunting): my global_id is 4528 2024-03-25T18:22:59.542 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.540+0000 7f4776ffd640 1 -- 172.21.15.97:0/4207727195 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4768004eb0 con 0x7f47881430c0 2024-03-25T18:22:59.548 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.547+0000 7f4776ffd640 10 client.?.objecter ms_handle_connect 0x7f47880a98f0 2024-03-25T18:22:59.548 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.547+0000 7f4776ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:59.550 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.548+0000 7f4776ffd640 1 -- 172.21.15.97:0/4207727195 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 214547532 0 0) 0x7f4768004eb0 con 0x7f47881430c0 2024-03-25T18:22:59.550 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.548+0000 7f4776ffd640 1 -- 172.21.15.97:0/4207727195 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4788136d90 con 0x7f47881430c0 2024-03-25T18:22:59.550 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.548+0000 7f4776ffd640 1 -- 172.21.15.97:0/4207727195 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f4780001f40 con 0x7f47881430c0 2024-03-25T18:22:59.550 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.548+0000 7f4776ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:59.551 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.548+0000 7f4776ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:22:59.551 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.548+0000 7f4776ffd640 10 monclient(hunting): dump: 2024-03-25T18:22:59.551 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:22:59.551 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:22:59.551 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:22:59.551 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:22:59.551 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:59.551 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:22:59.551 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:22:59.551 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:22:59.551 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:22:59.551 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:22:59.551 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.548+0000 7f4776ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:22:59.557 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.556+0000 7f4776ffd640 1 -- 172.21.15.97:0/4207727195 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 393280518 0 0) 0x7f4788136900 con 0x7f478813f710 2024-03-25T18:22:59.557 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.556+0000 7f4776ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:22:59.557 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.556+0000 7f4776ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:59.557 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.556+0000 7f4776ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:59.557 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.556+0000 7f4776ffd640 10 monclient(hunting): my global_id is 4521 2024-03-25T18:22:59.557 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.556+0000 7f4776ffd640 1 -- 172.21.15.97:0/4207727195 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4768004eb0 con 0x7f478813f710 2024-03-25T18:22:59.563 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.561+0000 7f4776ffd640 1 -- 172.21.15.97:0/4207727195 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1573061739 0 0) 0x7f47881397b0 con 0x7f47880a98f0 2024-03-25T18:22:59.563 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.561+0000 7f4776ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:22:59.563 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.561+0000 7f4776ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:59.563 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.561+0000 7f4776ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:59.563 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.561+0000 7f4776ffd640 10 monclient(hunting): my global_id is 4520 2024-03-25T18:22:59.564 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.561+0000 7f4776ffd640 1 -- 172.21.15.97:0/4207727195 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4788136900 con 0x7f47880a98f0 2024-03-25T18:22:59.564 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.561+0000 7f4776ffd640 1 -- 172.21.15.97:0/4207727195 <== mon.1 v1:172.21.15.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1510474301 0 0) 0x7f4788136d90 con 0x7f47881430c0 2024-03-25T18:22:59.564 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.561+0000 7f4776ffd640 1 -- 172.21.15.97:0/4207727195 >> v1:172.21.15.97:6790/0 conn(0x7f47880a98f0 legacy=0x7f47881390a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:59.564 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.561+0000 7f4776ffd640 1 -- 172.21.15.97:0/4207727195 >> v1:172.21.15.97:6789/0 conn(0x7f478813f710 legacy=0x7f47881370c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:59.564 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.561+0000 7f4776ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:22:59.564 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.561+0000 7f4776ffd640 1 monclient: found mon.b 2024-03-25T18:22:59.564 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.561+0000 7f4776ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:22:59.564 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.561+0000 7f4776ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:22:59.564 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.561+0000 7f4776ffd640 1 -- 172.21.15.97:0/4207727195 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4788139940 con 0x7f47881430c0 2024-03-25T18:22:59.564 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.561+0000 7f4776ffd640 10 monclient: _finish_auth 0 2024-03-25T18:22:59.564 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.561+0000 7f4776ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:22:59.565 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.561+0000 7f4776ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:22:59.565 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.561+0000 7f478cee6640 5 monclient: authenticate success, global_id 4528 2024-03-25T18:22:59.565 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.561+0000 7f478cee6640 10 monclient: _renew_subs 2024-03-25T18:22:59.565 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.561+0000 7f478cee6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:22:59.565 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.561+0000 7f478cee6640 1 -- 172.21.15.97:0/4207727195 --> v1:172.21.15.98:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47881377d0 con 0x7f47881430c0 2024-03-25T18:22:59.565 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.561+0000 7f478cee6640 10 client.4528.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:22:59.565 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.561+0000 7f478cee6640 10 monclient: _renew_subs 2024-03-25T18:22:59.565 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.561+0000 7f478cee6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:22:59.565 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.561+0000 7f478cee6640 1 -- 172.21.15.97:0/4207727195 --> v1:172.21.15.98:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f478813bff0 con 0x7f47881430c0 2024-03-25T18:22:59.565 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.561+0000 7f478cee6640 1 librados: init done 2024-03-25T18:22:59.566 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.562+0000 7f478cee6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:59.566 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.562+0000 7f478cee6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:59.566 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.562+0000 7f478cee6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:22:59.566 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.562+0000 7f478cee6640 1 -- 172.21.15.97:0/4207727195 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47880627b0 con 0x7f47881430c0 2024-03-25T18:22:59.794 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.792+0000 7f4776ffd640 1 -- 172.21.15.97:0/4207727195 <== mon.1 v1:172.21.15.98:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4780004e10 con 0x7f47881430c0 2024-03-25T18:22:59.794 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.792+0000 7f4776ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:22:59.794 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.792+0000 7f4776ffd640 1 -- 172.21.15.97:0/4207727195 <== mon.1 v1:172.21.15.98:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f4780005120 con 0x7f47881430c0 2024-03-25T18:22:59.794 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.792+0000 7f4776ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:59.794 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.792+0000 7f4776ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:22:59.794 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.792+0000 7f4776ffd640 10 monclient: dump: 2024-03-25T18:22:59.795 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:22:59.795 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:22:59.795 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:22:59.795 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:22:59.795 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:59.795 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:22:59.795 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:22:59.795 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:22:59.795 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:22:59.795 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:22:59.795 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.792+0000 7f4784d99640 4 set_mon_vals no callback set 2024-03-25T18:22:59.849 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.848+0000 7f4776ffd640 1 -- 172.21.15.97:0/4207727195 <== mon.1 v1:172.21.15.98:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2377622137 0 0) 0x7f478001b0f0 con 0x7f47881430c0 2024-03-25T18:22:59.850 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.849+0000 7f478659c640 10 monclient: get_auth_request con 0x7f476806de80 auth_method 0 2024-03-25T18:22:59.850 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.849+0000 7f4776ffd640 10 client.4528.objecter ms_handle_connect 0x7f476806de80 2024-03-25T18:22:59.855 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.854+0000 7f4776ffd640 1 -- 172.21.15.97:0/4207727195 <== mon.1 v1:172.21.15.98:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 430711918 0 0) 0x7f4780054c00 con 0x7f47881430c0 2024-03-25T18:22:59.855 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.854+0000 7f4776ffd640 10 client.4528.objecter ms_dispatch 0x7f4788000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:22:59.855 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.854+0000 7f4776ffd640 3 client.4528.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:22:59.855 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.854+0000 7f4776ffd640 3 client.4528.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:22:59.855 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.854+0000 7f4776ffd640 20 client.4528.objecter dump_active .. 0 homeless 2024-03-25T18:22:59.857 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.856+0000 7f4776ffd640 1 -- 172.21.15.97:0/4207727195 <== mon.1 v1:172.21.15.98:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4780050710 con 0x7f47881430c0 2024-03-25T18:22:59.857 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.856+0000 7f4776ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:59.858 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:22:59.856+0000 7f4776ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:23:00.099 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:00.097+0000 7f478cee6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:00.099 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:00.097+0000 7f478cee6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:00.099 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:00.097+0000 7f478cee6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:23:00.099 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:00.097+0000 7f478cee6640 1 -- 172.21.15.97:0/4207727195 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4788069f40 con 0x7f47881430c0 2024-03-25T18:23:00.120 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:00.118+0000 7f4776ffd640 1 -- 172.21.15.97:0/4207727195 <== mon.1 v1:172.21.15.98:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2169427412) 0x7f4780052640 con 0x7f47881430c0 2024-03-25T18:23:00.120 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:00.118+0000 7f4776ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:00.120 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:00.118+0000 7f4776ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:23:00.120 INFO:teuthology.misc.health.smithi097.stdout: 2024-03-25T18:23:00.120 INFO:teuthology.misc.health.smithi097.stdout:{"epoch":10,"fsid":"4c3d472c-79b8-486b-9e3d-2738d4306754","created":"2024-03-25T18:16:14.737309+0000","modified":"2024-03-25T18:16:20.357492+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:20.357492+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cb5152b9-262f-45c3-94b8-18d0e81df558","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6daf8793-0838-473d-a7d6-9a0d2894c75d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"136f1493-3b9c-4f43-a2e7-641482d69dae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"397d538c-4f64-4daf-9e8b-662d09f91327","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4f1df08-663f-4bda-93f8-3878ab7c24b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c4b18d70-a3ce-4b15-9b70-eae5525bd761","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8dd4033a-de0e-4f8a-96dc-59f3c1f825b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0077722e-ffa6-4a3d-9891-929d9869b286","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:23:00.122 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:00.120+0000 7f478cee6640 10 librados: watch_flush enter 2024-03-25T18:23:00.123 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:00.120+0000 7f478cee6640 10 librados: watch_flush exit 2024-03-25T18:23:00.123 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:00.120+0000 7f478cee6640 20 client.4528.objecter shutdown clearing up homeless session... 2024-03-25T18:23:00.123 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:00.120+0000 7f478cee6640 10 client.4528.objecter successfully canceled tick 2024-03-25T18:23:00.123 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:00.120+0000 7f478cee6640 1 -- 172.21.15.97:0/4207727195 >> v1:172.21.15.98:6800/800387573 conn(0x7f476806de80 legacy=0x7f4768070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:00.123 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:00.120+0000 7f478cee6640 10 monclient: shutdown 2024-03-25T18:23:00.123 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:00.120+0000 7f478cee6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:23:00.123 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:00.120+0000 7f478cee6640 1 -- 172.21.15.97:0/4207727195 >> v1:172.21.15.98:6789/0 conn(0x7f47881430c0 legacy=0x7f47881e1e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:00.123 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:00.120+0000 7f478cee6640 1 -- 172.21.15.97:0/4207727195 shutdown_connections 2024-03-25T18:23:00.123 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:00.120+0000 7f478cee6640 1 -- 172.21.15.97:0/4207727195 >> 172.21.15.97:0/4207727195 conn(0x7f47880b7ea0 msgr2=0x7f4788074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:23:00.123 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:00.120+0000 7f478cee6640 1 -- 172.21.15.97:0/4207727195 shutdown_connections 2024-03-25T18:23:00.123 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:00.120+0000 7f478cee6640 1 -- 172.21.15.97:0/4207727195 wait complete. 2024-03-25T18:23:00.123 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:00.121+0000 7f478cee6640 1 librados: shutdown 2024-03-25T18:23:00.135 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:23:06.943 DEBUG:teuthology.orchestra.run.smithi097:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:23:07.059 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.058+0000 7f5f02d58640 10 monclient: get_monmap_and_config 2024-03-25T18:23:07.059 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.058+0000 7f5f02d58640 10 monclient: build_initial_monmap 2024-03-25T18:23:07.059 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.058+0000 7f5f02d58640 10 monclient: monmap: 2024-03-25T18:23:07.059 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:23:07.059 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:23:07.060 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:23:07.059425+0000 2024-03-25T18:23:07.060 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:23:07.059425+0000 2024-03-25T18:23:07.060 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:23:07.060 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:23:07.060 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:23:07.060 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T18:23:07.060 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-b 2024-03-25T18:23:07.060 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:23:07.060 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.059+0000 7f5f02d58640 1 Processor -- start 2024-03-25T18:23:07.060 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.059+0000 7f5f02d58640 1 -- start start 2024-03-25T18:23:07.061 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.059+0000 7f5f02d58640 10 monclient: init 2024-03-25T18:23:07.061 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.059+0000 7f5f02d58640 10 monclient: _reopen_session rank -1 2024-03-25T18:23:07.061 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.059+0000 7f5f02d58640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:23:07.062 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.059+0000 7f5f02d58640 10 monclient(hunting): picked mon.noname-c con 0x7f5efc0a98f0 addr v1:172.21.15.98:6789/0 2024-03-25T18:23:07.062 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.059+0000 7f5f02d58640 10 monclient(hunting): picked mon.noname-b con 0x7f5efc13f710 addr v1:172.21.15.97:6790/0 2024-03-25T18:23:07.062 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.060+0000 7f5f02d58640 10 monclient(hunting): picked mon.noname-a con 0x7f5efc1430c0 addr v1:172.21.15.97:6789/0 2024-03-25T18:23:07.062 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.060+0000 7f5f02d58640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5efc142520 con 0x7f5efc1430c0 2024-03-25T18:23:07.062 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.060+0000 7f5f02d58640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5efc0b28a0 con 0x7f5efc0a98f0 2024-03-25T18:23:07.062 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.060+0000 7f5f02d58640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5efc0b2a30 con 0x7f5efc13f710 2024-03-25T18:23:07.062 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.060+0000 7f5f02d58640 10 monclient(hunting): _renew_subs 2024-03-25T18:23:07.062 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.060+0000 7f5f02d58640 10 monclient(hunting): authenticate will time out at 7999.336070s 2024-03-25T18:23:07.079 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.077+0000 7f5f012f5640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f5efc1430c0 0x7f5efc14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:34306/0 (socket says 172.21.15.97:34306) 2024-03-25T18:23:07.079 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.078+0000 7f5f012f5640 1 -- 172.21.15.97:0/825192735 learned_addr learned my addr 172.21.15.97:0/825192735 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:23:07.092 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.090+0000 7f5efb7fe640 1 -- 172.21.15.97:0/825192735 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1597563880 0 0) 0x7f5efc0b28a0 con 0x7f5efc0a98f0 2024-03-25T18:23:07.092 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.090+0000 7f5efb7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:23:07.092 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.090+0000 7f5efb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:07.092 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.090+0000 7f5efb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:07.092 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.090+0000 7f5efb7fe640 10 monclient(hunting): my global_id is 4531 2024-03-25T18:23:07.092 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.090+0000 7f5efb7fe640 1 -- 172.21.15.97:0/825192735 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5eec003520 con 0x7f5efc0a98f0 2024-03-25T18:23:07.101 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.100+0000 7f5efb7fe640 1 -- 172.21.15.97:0/825192735 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 354594652 0 0) 0x7f5efc0b2a30 con 0x7f5efc13f710 2024-03-25T18:23:07.101 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.100+0000 7f5efb7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:23:07.101 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.100+0000 7f5efb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:07.101 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.100+0000 7f5efb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:07.101 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.100+0000 7f5efb7fe640 10 monclient(hunting): my global_id is 4523 2024-03-25T18:23:07.102 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.100+0000 7f5efb7fe640 1 -- 172.21.15.97:0/825192735 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5efc0b28a0 con 0x7f5efc13f710 2024-03-25T18:23:07.103 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.101+0000 7f5efb7fe640 1 -- 172.21.15.97:0/825192735 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2739473814 0 0) 0x7f5eec003520 con 0x7f5efc0a98f0 2024-03-25T18:23:07.103 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.101+0000 7f5efb7fe640 1 -- 172.21.15.97:0/825192735 >> v1:172.21.15.97:6790/0 conn(0x7f5efc13f710 legacy=0x7f5efc1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:07.103 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.101+0000 7f5efb7fe640 1 -- 172.21.15.97:0/825192735 >> v1:172.21.15.97:6789/0 conn(0x7f5efc1430c0 legacy=0x7f5efc14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:07.103 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.101+0000 7f5efb7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:23:07.103 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.101+0000 7f5efb7fe640 1 monclient: found mon.noname-c 2024-03-25T18:23:07.103 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.101+0000 7f5efb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:23:07.104 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.101+0000 7f5efb7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.98:6789/0 2024-03-25T18:23:07.104 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.101+0000 7f5efb7fe640 1 -- 172.21.15.97:0/825192735 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5efc0b44c0 con 0x7f5efc0a98f0 2024-03-25T18:23:07.104 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.101+0000 7f5efb7fe640 10 monclient: _finish_auth 0 2024-03-25T18:23:07.104 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.101+0000 7f5efb7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:23:07.104 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.101+0000 7f5efb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:23:07.104 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.101+0000 7f5efb7fe640 1 -- 172.21.15.97:0/825192735 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f5ef0001690 con 0x7f5efc0a98f0 2024-03-25T18:23:07.104 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.101+0000 7f5efb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:07.104 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.101+0000 7f5efb7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:23:07.104 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.101+0000 7f5efb7fe640 10 monclient: dump: 2024-03-25T18:23:07.104 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:23:07.104 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:23:07.104 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:23:07.104 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:23:07.104 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:07.104 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:23:07.105 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:23:07.105 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:23:07.105 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:23:07.105 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:23:07.105 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.101+0000 7f5f02d58640 5 monclient: authenticate success, global_id 4531 2024-03-25T18:23:07.105 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.101+0000 7f5f02d58640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:23:07.111 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.109+0000 7f5efb7fe640 1 -- 172.21.15.97:0/825192735 <== mon.1 v1:172.21.15.98:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5ef00017f0 con 0x7f5efc0a98f0 2024-03-25T18:23:07.111 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.109+0000 7f5efb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:23:07.111 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.109+0000 7f5efb7fe640 1 -- 172.21.15.97:0/825192735 <== mon.1 v1:172.21.15.98:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f5ef0001c00 con 0x7f5efc0a98f0 2024-03-25T18:23:07.111 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.109+0000 7f5f02d58640 10 monclient: get_monmap_and_config success 2024-03-25T18:23:07.111 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.109+0000 7f5f02d58640 4 set_mon_vals no callback set 2024-03-25T18:23:07.111 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.109+0000 7f5efb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:07.111 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.109+0000 7f5efb7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:23:07.111 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.109+0000 7f5efb7fe640 10 monclient: dump: 2024-03-25T18:23:07.111 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:23:07.111 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:23:07.111 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:23:07.111 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:23:07.112 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:07.112 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:23:07.112 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:23:07.112 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:23:07.112 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:23:07.112 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:23:07.113 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.110+0000 7f5f02d58640 10 monclient: shutdown 2024-03-25T18:23:07.113 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.110+0000 7f5f02d58640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:23:07.113 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.110+0000 7f5f02d58640 1 -- 172.21.15.97:0/825192735 >> v1:172.21.15.98:6789/0 conn(0x7f5efc0a98f0 legacy=0x7f5efc0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:07.113 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.110+0000 7f5f02d58640 1 -- 172.21.15.97:0/825192735 shutdown_connections 2024-03-25T18:23:07.113 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.110+0000 7f5f02d58640 1 -- 172.21.15.97:0/825192735 >> 172.21.15.97:0/825192735 conn(0x7f5efc0b7ea0 msgr2=0x7f5efc0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:23:07.113 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.110+0000 7f5f02d58640 1 -- 172.21.15.97:0/825192735 shutdown_connections 2024-03-25T18:23:07.113 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.110+0000 7f5f02d58640 1 -- 172.21.15.97:0/825192735 wait complete. 2024-03-25T18:23:07.113 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.111+0000 7f5f02d58640 10 monclient: build_initial_monmap 2024-03-25T18:23:07.113 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.111+0000 7f5f02d58640 10 monclient: monmap: 2024-03-25T18:23:07.114 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:23:07.114 INFO:teuthology.misc.health.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:23:07.114 INFO:teuthology.misc.health.smithi097.stderr:last_changed 0.000000 2024-03-25T18:23:07.114 INFO:teuthology.misc.health.smithi097.stderr:created 0.000000 2024-03-25T18:23:07.114 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:23:07.114 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 1 2024-03-25T18:23:07.114 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:23:07.114 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T18:23:07.114 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-c 2024-03-25T18:23:07.114 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:23:07.114 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.111+0000 7f5f02d58640 1 librados: starting msgr at 2024-03-25T18:23:07.114 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.111+0000 7f5f02d58640 1 librados: starting objecter 2024-03-25T18:23:07.114 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.111+0000 7f5f02d58640 1 Processor -- start 2024-03-25T18:23:07.114 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.111+0000 7f5f02d58640 1 -- start start 2024-03-25T18:23:07.115 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.111+0000 7f5f02d58640 1 librados: setting wanted keys 2024-03-25T18:23:07.115 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.111+0000 7f5f02d58640 1 librados: calling monclient init 2024-03-25T18:23:07.115 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.111+0000 7f5f02d58640 10 monclient: init 2024-03-25T18:23:07.116 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.112+0000 7f5f02d58640 10 monclient: _reopen_session rank -1 2024-03-25T18:23:07.116 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.112+0000 7f5f02d58640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:23:07.116 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.112+0000 7f5f02d58640 10 monclient(hunting): picked mon.noname-a con 0x7f5efc0a98f0 addr v1:172.21.15.97:6789/0 2024-03-25T18:23:07.116 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.112+0000 7f5f02d58640 10 monclient(hunting): picked mon.noname-c con 0x7f5efc13f710 addr v1:172.21.15.97:6790/0 2024-03-25T18:23:07.116 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.112+0000 7f5f02d58640 10 monclient(hunting): picked mon.noname-b con 0x7f5efc1430c0 addr v1:172.21.15.98:6789/0 2024-03-25T18:23:07.116 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.112+0000 7f5f02d58640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5efc139a40 con 0x7f5efc0a98f0 2024-03-25T18:23:07.116 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.112+0000 7f5f02d58640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5efc133f20 con 0x7f5efc1430c0 2024-03-25T18:23:07.116 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.112+0000 7f5f02d58640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5efc0ab950 con 0x7f5efc13f710 2024-03-25T18:23:07.116 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.112+0000 7f5f02d58640 10 monclient(hunting): _renew_subs 2024-03-25T18:23:07.116 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.112+0000 7f5f02d58640 10 monclient(hunting): authenticate will time out at 7999.388672s 2024-03-25T18:23:07.127 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.126+0000 7f5f00af4640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f5efc0a98f0 0x7f5efc138c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:34356/0 (socket says 172.21.15.97:34356) 2024-03-25T18:23:07.128 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.126+0000 7f5f00af4640 1 -- 172.21.15.97:0/1245124952 learned_addr learned my addr 172.21.15.97:0/1245124952 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:23:07.128 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.127+0000 7f5ef97fa640 10 client.?.objecter ms_handle_connect 0x7f5efc0a98f0 2024-03-25T18:23:07.128 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.127+0000 7f5ef97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:07.139 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.137+0000 7f5ef97fa640 1 -- 172.21.15.97:0/1245124952 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2480396259 0 0) 0x7f5efc139a40 con 0x7f5efc0a98f0 2024-03-25T18:23:07.139 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.137+0000 7f5ef97fa640 10 monclient(hunting): _negotiate 2024-03-25T18:23:07.139 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.137+0000 7f5ef97fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:07.139 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.137+0000 7f5ef97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:07.139 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.137+0000 7f5ef97fa640 10 monclient(hunting): my global_id is 4527 2024-03-25T18:23:07.139 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.137+0000 7f5ef97fa640 1 -- 172.21.15.97:0/1245124952 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ee0004eb0 con 0x7f5efc0a98f0 2024-03-25T18:23:07.143 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.142+0000 7f5ef97fa640 10 client.?.objecter ms_handle_connect 0x7f5efc1430c0 2024-03-25T18:23:07.143 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.142+0000 7f5ef97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:07.152 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.151+0000 7f5ef97fa640 10 client.?.objecter ms_handle_connect 0x7f5efc13f710 2024-03-25T18:23:07.152 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.151+0000 7f5ef97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:07.161 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.160+0000 7f5ef97fa640 1 -- 172.21.15.97:0/1245124952 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3396298100 0 0) 0x7f5ee0004eb0 con 0x7f5efc0a98f0 2024-03-25T18:23:07.161 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.160+0000 7f5ef97fa640 1 -- 172.21.15.97:0/1245124952 --> v1:172.21.15.97:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5efc139a40 con 0x7f5efc0a98f0 2024-03-25T18:23:07.162 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.160+0000 7f5ef97fa640 1 -- 172.21.15.97:0/1245124952 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f5ef0001690 con 0x7f5efc0a98f0 2024-03-25T18:23:07.162 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.160+0000 7f5ef97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:07.162 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.160+0000 7f5ef97fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:23:07.162 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.160+0000 7f5ef97fa640 10 monclient(hunting): dump: 2024-03-25T18:23:07.162 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:23:07.162 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:23:07.162 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:23:07.162 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:23:07.162 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:07.162 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:23:07.162 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:23:07.162 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:23:07.162 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:23:07.162 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:23:07.163 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.160+0000 7f5ef97fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:23:07.172 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.170+0000 7f5ef97fa640 1 -- 172.21.15.97:0/1245124952 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 569133419 0 0) 0x7f5efc133f20 con 0x7f5efc1430c0 2024-03-25T18:23:07.172 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.170+0000 7f5ef97fa640 10 monclient(hunting): _negotiate 2024-03-25T18:23:07.172 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.170+0000 7f5ef97fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:07.172 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.170+0000 7f5ef97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:07.172 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.170+0000 7f5ef97fa640 10 monclient(hunting): my global_id is 4534 2024-03-25T18:23:07.172 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.170+0000 7f5ef97fa640 1 -- 172.21.15.97:0/1245124952 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ee0004eb0 con 0x7f5efc1430c0 2024-03-25T18:23:07.173 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.170+0000 7f5ef97fa640 1 -- 172.21.15.97:0/1245124952 <== mon.0 v1:172.21.15.97:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 145162955 0 0) 0x7f5efc139a40 con 0x7f5efc0a98f0 2024-03-25T18:23:07.173 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.170+0000 7f5ef97fa640 1 -- 172.21.15.97:0/1245124952 >> v1:172.21.15.97:6790/0 conn(0x7f5efc13f710 legacy=0x7f5efc133100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:07.173 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.170+0000 7f5ef97fa640 1 -- 172.21.15.97:0/1245124952 >> v1:172.21.15.98:6789/0 conn(0x7f5efc1430c0 legacy=0x7f5efc133810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:07.173 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.170+0000 7f5ef97fa640 10 monclient: _finish_hunting 0 2024-03-25T18:23:07.173 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.170+0000 7f5ef97fa640 1 monclient: found mon.a 2024-03-25T18:23:07.173 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.170+0000 7f5ef97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:23:07.173 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.170+0000 7f5ef97fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:23:07.173 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.170+0000 7f5ef97fa640 1 -- 172.21.15.97:0/1245124952 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5efc0abc30 con 0x7f5efc0a98f0 2024-03-25T18:23:07.173 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.170+0000 7f5ef97fa640 10 monclient: _finish_auth 0 2024-03-25T18:23:07.173 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.170+0000 7f5ef97fa640 10 monclient: _check_auth_tickets 2024-03-25T18:23:07.173 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.170+0000 7f5ef97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:23:07.173 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.170+0000 7f5f02d58640 5 monclient: authenticate success, global_id 4527 2024-03-25T18:23:07.173 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.170+0000 7f5f02d58640 10 monclient: _renew_subs 2024-03-25T18:23:07.173 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.170+0000 7f5f02d58640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:23:07.174 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.170+0000 7f5f02d58640 1 -- 172.21.15.97:0/1245124952 --> v1:172.21.15.97:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5efc0abf10 con 0x7f5efc0a98f0 2024-03-25T18:23:07.174 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.170+0000 7f5f02d58640 10 client.4527.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:23:07.174 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.170+0000 7f5f02d58640 10 monclient: _renew_subs 2024-03-25T18:23:07.174 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.170+0000 7f5f02d58640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:23:07.174 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.170+0000 7f5f02d58640 1 -- 172.21.15.97:0/1245124952 --> v1:172.21.15.97:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5efc139370 con 0x7f5efc0a98f0 2024-03-25T18:23:07.174 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.170+0000 7f5f02d58640 1 librados: init done 2024-03-25T18:23:07.174 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.171+0000 7f5f02d58640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:07.174 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.171+0000 7f5f02d58640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:07.175 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.171+0000 7f5f02d58640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:23:07.175 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.171+0000 7f5f02d58640 1 -- 172.21.15.97:0/1245124952 --> v1:172.21.15.97:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5efc138980 con 0x7f5efc0a98f0 2024-03-25T18:23:07.462 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.460+0000 7f5ef97fa640 1 -- 172.21.15.97:0/1245124952 <== mon.0 v1:172.21.15.97:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5ef00038a0 con 0x7f5efc0a98f0 2024-03-25T18:23:07.462 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.460+0000 7f5ef97fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:23:07.462 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.460+0000 7f5ef97fa640 1 -- 172.21.15.97:0/1245124952 <== mon.0 v1:172.21.15.97:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f5ef0003ad0 con 0x7f5efc0a98f0 2024-03-25T18:23:07.463 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.460+0000 7f5ef97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:07.463 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.460+0000 7f5ef97fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:23:07.463 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.460+0000 7f5ef97fa640 10 monclient: dump: 2024-03-25T18:23:07.463 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:23:07.463 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:23:07.463 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:23:07.463 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:23:07.463 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:07.463 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:23:07.463 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:23:07.463 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:23:07.464 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:23:07.464 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:23:07.464 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.460+0000 7f5efaffd640 4 set_mon_vals no callback set 2024-03-25T18:23:07.464 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.461+0000 7f5ef97fa640 1 -- 172.21.15.97:0/1245124952 <== mon.0 v1:172.21.15.97:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2377622137 0 0) 0x7f5ef0003cf0 con 0x7f5efc0a98f0 2024-03-25T18:23:07.464 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.462+0000 7f5efbfff640 10 monclient: get_auth_request con 0x7f5ee006dd90 auth_method 0 2024-03-25T18:23:07.465 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.462+0000 7f5ef97fa640 10 client.4527.objecter ms_handle_connect 0x7f5ee006dd90 2024-03-25T18:23:07.469 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.468+0000 7f5ef97fa640 1 -- 172.21.15.97:0/1245124952 <== mon.0 v1:172.21.15.97:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 430711918 0 0) 0x7f5ef0054760 con 0x7f5efc0a98f0 2024-03-25T18:23:07.470 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.468+0000 7f5ef97fa640 10 client.4527.objecter ms_dispatch 0x7f5efc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:23:07.470 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.468+0000 7f5ef97fa640 3 client.4527.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:23:07.470 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.468+0000 7f5ef97fa640 3 client.4527.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:23:07.470 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.468+0000 7f5ef97fa640 20 client.4527.objecter dump_active .. 0 homeless 2024-03-25T18:23:07.470 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.468+0000 7f5ef97fa640 1 -- 172.21.15.97:0/1245124952 <== mon.0 v1:172.21.15.97:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5ef0050270 con 0x7f5efc0a98f0 2024-03-25T18:23:07.470 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.468+0000 7f5ef97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:07.470 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.468+0000 7f5ef97fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:23:07.720 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.719+0000 7f5f02d58640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:07.721 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.719+0000 7f5f02d58640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:07.721 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.719+0000 7f5f02d58640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:23:07.721 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.719+0000 7f5f02d58640 1 -- 172.21.15.97:0/1245124952 --> v1:172.21.15.97:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5efc0627b0 con 0x7f5efc0a98f0 2024-03-25T18:23:07.791 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.789+0000 7f5ef97fa640 1 -- 172.21.15.97:0/1245124952 <== mon.0 v1:172.21.15.97:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2169427412) 0x7f5ef00521a0 con 0x7f5efc0a98f0 2024-03-25T18:23:07.791 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.789+0000 7f5ef97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:07.791 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.789+0000 7f5ef97fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:23:07.791 INFO:teuthology.misc.health.smithi097.stdout: 2024-03-25T18:23:07.791 INFO:teuthology.misc.health.smithi097.stdout:{"epoch":10,"fsid":"4c3d472c-79b8-486b-9e3d-2738d4306754","created":"2024-03-25T18:16:14.737309+0000","modified":"2024-03-25T18:16:20.357492+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:20.357492+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cb5152b9-262f-45c3-94b8-18d0e81df558","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6daf8793-0838-473d-a7d6-9a0d2894c75d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"136f1493-3b9c-4f43-a2e7-641482d69dae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"397d538c-4f64-4daf-9e8b-662d09f91327","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4f1df08-663f-4bda-93f8-3878ab7c24b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c4b18d70-a3ce-4b15-9b70-eae5525bd761","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8dd4033a-de0e-4f8a-96dc-59f3c1f825b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0077722e-ffa6-4a3d-9891-929d9869b286","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:23:07.794 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.792+0000 7f5f02d58640 10 librados: watch_flush enter 2024-03-25T18:23:07.794 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.792+0000 7f5f02d58640 10 librados: watch_flush exit 2024-03-25T18:23:07.794 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.792+0000 7f5f02d58640 20 client.4527.objecter shutdown clearing up homeless session... 2024-03-25T18:23:07.794 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.792+0000 7f5f02d58640 10 client.4527.objecter successfully canceled tick 2024-03-25T18:23:07.794 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.792+0000 7f5f02d58640 1 -- 172.21.15.97:0/1245124952 >> v1:172.21.15.98:6800/800387573 conn(0x7f5ee006dd90 legacy=0x7f5ee0070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:07.794 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.792+0000 7f5f02d58640 10 monclient: shutdown 2024-03-25T18:23:07.794 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.792+0000 7f5f02d58640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:23:07.794 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.792+0000 7f5f02d58640 1 -- 172.21.15.97:0/1245124952 >> v1:172.21.15.97:6789/0 conn(0x7f5efc0a98f0 legacy=0x7f5efc138c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:07.794 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.792+0000 7f5f02d58640 1 -- 172.21.15.97:0/1245124952 shutdown_connections 2024-03-25T18:23:07.794 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.792+0000 7f5f02d58640 1 -- 172.21.15.97:0/1245124952 >> 172.21.15.97:0/1245124952 conn(0x7f5efc0b7ea0 msgr2=0x7f5efc074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:23:07.794 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.792+0000 7f5f02d58640 1 -- 172.21.15.97:0/1245124952 shutdown_connections 2024-03-25T18:23:07.795 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.792+0000 7f5f02d58640 1 -- 172.21.15.97:0/1245124952 wait complete. 2024-03-25T18:23:07.795 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:07.792+0000 7f5f02d58640 1 librados: shutdown 2024-03-25T18:23:07.807 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:23:13.206 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:07:15.244 34931== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:13.216 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:07:15.255 34931== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:13.250 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:07:15.289 34931== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:13.255 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:07:15.298 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:13.265 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:07:15.308 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:13.276 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:07:15.315 34935== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:13.280 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:07:15.319 34935== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:13.284 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:07:15.327 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:13.303 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:07:15.342 34935== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:14.613 DEBUG:teuthology.orchestra.run.smithi097:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:23:14.729 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:14.728+0000 7f403bdc6640 10 monclient: get_monmap_and_config 2024-03-25T18:23:14.729 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:14.728+0000 7f403bdc6640 10 monclient: build_initial_monmap 2024-03-25T18:23:14.729 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:14.728+0000 7f403bdc6640 10 monclient: monmap: 2024-03-25T18:23:14.729 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:23:14.729 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:23:14.730 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:23:14.729377+0000 2024-03-25T18:23:14.730 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:23:14.729377+0000 2024-03-25T18:23:14.730 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:23:14.730 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:23:14.730 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:23:14.730 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T18:23:14.730 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-b 2024-03-25T18:23:14.730 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:23:14.731 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:14.728+0000 7f403bdc6640 1 Processor -- start 2024-03-25T18:23:14.731 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:14.728+0000 7f403bdc6640 1 -- start start 2024-03-25T18:23:14.731 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:14.728+0000 7f403bdc6640 10 monclient: init 2024-03-25T18:23:14.731 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:14.729+0000 7f403bdc6640 10 monclient: _reopen_session rank -1 2024-03-25T18:23:14.731 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:14.729+0000 7f403bdc6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:23:14.731 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:14.729+0000 7f403bdc6640 10 monclient(hunting): picked mon.noname-a con 0x7f40340a98f0 addr v1:172.21.15.97:6789/0 2024-03-25T18:23:14.731 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:14.729+0000 7f403bdc6640 10 monclient(hunting): picked mon.noname-b con 0x7f403413f710 addr v1:172.21.15.97:6790/0 2024-03-25T18:23:14.731 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:14.729+0000 7f403bdc6640 10 monclient(hunting): picked mon.noname-c con 0x7f40341430c0 addr v1:172.21.15.98:6789/0 2024-03-25T18:23:14.731 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:14.729+0000 7f403bdc6640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4034142520 con 0x7f40340a98f0 2024-03-25T18:23:14.732 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:14.729+0000 7f403bdc6640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40340b28a0 con 0x7f40341430c0 2024-03-25T18:23:14.732 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:14.729+0000 7f403bdc6640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40340b2a30 con 0x7f403413f710 2024-03-25T18:23:14.732 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:14.729+0000 7f403bdc6640 10 monclient(hunting): _renew_subs 2024-03-25T18:23:14.732 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:14.729+0000 7f403bdc6640 10 monclient(hunting): authenticate will time out at 8007.005603s 2024-03-25T18:23:14.740 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:14.739+0000 7f403a363640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f40341430c0 0x7f403414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.97:53434/0 (socket says 172.21.15.97:53434) 2024-03-25T18:23:14.740 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:14.739+0000 7f403a363640 1 -- 172.21.15.97:0/2951641181 learned_addr learned my addr 172.21.15.97:0/2951641181 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:23:14.748 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:14.747+0000 7f4038b60640 1 -- 172.21.15.97:0/2951641181 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 352733902 0 0) 0x7f4034142520 con 0x7f40340a98f0 2024-03-25T18:23:14.749 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:14.747+0000 7f4038b60640 10 monclient(hunting): _negotiate 2024-03-25T18:23:14.749 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:14.747+0000 7f4038b60640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:14.749 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:14.747+0000 7f4038b60640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:14.749 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:14.747+0000 7f4038b60640 10 monclient(hunting): my global_id is 4530 2024-03-25T18:23:14.749 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:14.747+0000 7f4038b60640 1 -- 172.21.15.97:0/2951641181 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4020003520 con 0x7f40340a98f0 2024-03-25T18:23:14.758 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:14.757+0000 7f4038b60640 1 -- 172.21.15.97:0/2951641181 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2178078925 0 0) 0x7f4020003520 con 0x7f40340a98f0 2024-03-25T18:23:14.759 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:14.757+0000 7f4038b60640 1 -- 172.21.15.97:0/2951641181 >> v1:172.21.15.97:6790/0 conn(0x7f403413f710 legacy=0x7f40341429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:14.759 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:14.757+0000 7f4038b60640 1 -- 172.21.15.97:0/2951641181 >> v1:172.21.15.98:6789/0 conn(0x7f40341430c0 legacy=0x7f403414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:14.759 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:14.757+0000 7f4038b60640 10 monclient: _finish_hunting 0 2024-03-25T18:23:14.759 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:14.757+0000 7f4038b60640 1 monclient: found mon.noname-a 2024-03-25T18:23:14.759 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:14.757+0000 7f4038b60640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:23:14.759 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:14.757+0000 7f4038b60640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.97:6789/0 2024-03-25T18:23:14.759 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:14.757+0000 7f4038b60640 1 -- 172.21.15.97:0/2951641181 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40340b44c0 con 0x7f40340a98f0 2024-03-25T18:23:14.759 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:14.757+0000 7f4038b60640 10 monclient: _finish_auth 0 2024-03-25T18:23:14.759 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:14.757+0000 7f4038b60640 10 monclient: _check_auth_tickets 2024-03-25T18:23:14.759 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:14.757+0000 7f4038b60640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:23:14.759 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:14.757+0000 7f4038b60640 1 -- 172.21.15.97:0/2951641181 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f4030001690 con 0x7f40340a98f0 2024-03-25T18:23:14.759 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:14.757+0000 7f4038b60640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:14.759 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:14.757+0000 7f4038b60640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:23:14.760 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:14.757+0000 7f4038b60640 10 monclient: dump: 2024-03-25T18:23:14.760 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:23:14.760 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:23:14.760 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:23:14.760 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:23:14.760 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:14.760 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:23:14.760 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:23:14.760 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:23:14.760 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:23:14.760 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:23:14.760 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:14.757+0000 7f403bdc6640 5 monclient: authenticate success, global_id 4530 2024-03-25T18:23:14.760 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:14.757+0000 7f403bdc6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:23:14.780 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:14.778+0000 7f4038b60640 1 -- 172.21.15.97:0/2951641181 <== mon.0 v1:172.21.15.97:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f40300017f0 con 0x7f40340a98f0 2024-03-25T18:23:14.780 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:14.778+0000 7f4038b60640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:23:14.780 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:14.778+0000 7f4038b60640 1 -- 172.21.15.97:0/2951641181 <== mon.0 v1:172.21.15.97:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f4030001c40 con 0x7f40340a98f0 2024-03-25T18:23:14.780 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:14.778+0000 7f4038b60640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:14.780 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:14.778+0000 7f4038b60640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:23:14.780 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:14.778+0000 7f4038b60640 10 monclient: dump: 2024-03-25T18:23:14.780 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:23:14.780 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:23:14.780 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:23:14.780 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:23:14.780 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:14.780 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:23:14.781 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:23:14.781 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:23:14.781 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:23:14.781 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:23:14.781 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:14.778+0000 7f403bdc6640 10 monclient: get_monmap_and_config success 2024-03-25T18:23:14.781 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:14.778+0000 7f403bdc6640 4 set_mon_vals no callback set 2024-03-25T18:23:14.783 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:14.779+0000 7f403bdc6640 10 monclient: shutdown 2024-03-25T18:23:14.783 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:14.779+0000 7f403bdc6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:23:14.783 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:14.779+0000 7f403bdc6640 1 -- 172.21.15.97:0/2951641181 >> v1:172.21.15.97:6789/0 conn(0x7f40340a98f0 legacy=0x7f40340a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:14.783 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:14.779+0000 7f403bdc6640 1 -- 172.21.15.97:0/2951641181 shutdown_connections 2024-03-25T18:23:14.783 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:14.779+0000 7f403bdc6640 1 -- 172.21.15.97:0/2951641181 >> 172.21.15.97:0/2951641181 conn(0x7f40340b7ea0 msgr2=0x7f40340b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:23:14.784 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:14.779+0000 7f403bdc6640 1 -- 172.21.15.97:0/2951641181 shutdown_connections 2024-03-25T18:23:14.784 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:14.779+0000 7f403bdc6640 1 -- 172.21.15.97:0/2951641181 wait complete. 2024-03-25T18:23:14.784 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:14.780+0000 7f403bdc6640 10 monclient: build_initial_monmap 2024-03-25T18:23:14.784 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:14.780+0000 7f403bdc6640 10 monclient: monmap: 2024-03-25T18:23:14.784 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:23:14.784 INFO:teuthology.misc.health.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:23:14.784 INFO:teuthology.misc.health.smithi097.stderr:last_changed 0.000000 2024-03-25T18:23:14.784 INFO:teuthology.misc.health.smithi097.stderr:created 0.000000 2024-03-25T18:23:14.784 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:23:14.784 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 1 2024-03-25T18:23:14.784 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:23:14.784 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T18:23:14.784 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-c 2024-03-25T18:23:14.784 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:23:14.785 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:14.781+0000 7f403bdc6640 1 librados: starting msgr at 2024-03-25T18:23:14.785 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:14.781+0000 7f403bdc6640 1 librados: starting objecter 2024-03-25T18:23:14.785 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:14.781+0000 7f403bdc6640 1 Processor -- start 2024-03-25T18:23:14.785 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:14.781+0000 7f403bdc6640 1 -- start start 2024-03-25T18:23:14.785 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:14.781+0000 7f403bdc6640 1 librados: setting wanted keys 2024-03-25T18:23:14.785 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:14.781+0000 7f403bdc6640 1 librados: calling monclient init 2024-03-25T18:23:14.785 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:14.781+0000 7f403bdc6640 10 monclient: init 2024-03-25T18:23:14.785 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:14.781+0000 7f403bdc6640 10 monclient: _reopen_session rank -1 2024-03-25T18:23:14.785 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:14.781+0000 7f403bdc6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:23:14.785 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:14.781+0000 7f403bdc6640 10 monclient(hunting): picked mon.noname-b con 0x7f40340a98f0 addr v1:172.21.15.98:6789/0 2024-03-25T18:23:14.785 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:14.781+0000 7f403bdc6640 10 monclient(hunting): picked mon.noname-a con 0x7f403413f710 addr v1:172.21.15.97:6789/0 2024-03-25T18:23:14.785 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:14.782+0000 7f403bdc6640 10 monclient(hunting): picked mon.noname-c con 0x7f40341430c0 addr v1:172.21.15.97:6790/0 2024-03-25T18:23:14.785 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:14.782+0000 7f403bdc6640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4034138c70 con 0x7f403413f710 2024-03-25T18:23:14.785 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:14.782+0000 7f403bdc6640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40341367d0 con 0x7f40340a98f0 2024-03-25T18:23:14.785 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:14.782+0000 7f403bdc6640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4034136960 con 0x7f40341430c0 2024-03-25T18:23:14.786 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:14.782+0000 7f403bdc6640 10 monclient(hunting): _renew_subs 2024-03-25T18:23:14.786 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:14.782+0000 7f403bdc6640 10 monclient(hunting): authenticate will time out at 8007.058041s 2024-03-25T18:23:14.796 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:14.795+0000 7f403a363640 1 --1- >> v1:172.21.15.97:6790/0 conn(0x7f40341430c0 0x7f40341e2cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.97:6790/0 says I am v1:172.21.15.97:58130/0 (socket says 172.21.15.97:58130) 2024-03-25T18:23:14.796 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:14.795+0000 7f403a363640 1 -- 172.21.15.97:0/2661920998 learned_addr learned my addr 172.21.15.97:0/2661920998 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:23:14.797 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:14.796+0000 7f402a7fc640 10 client.?.objecter ms_handle_connect 0x7f40341430c0 2024-03-25T18:23:14.797 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:14.796+0000 7f402a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:14.802 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:14.801+0000 7f402a7fc640 10 client.?.objecter ms_handle_connect 0x7f403413f710 2024-03-25T18:23:14.802 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:14.801+0000 7f402a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:14.805 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:14.804+0000 7f402a7fc640 1 -- 172.21.15.97:0/2661920998 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2694996043 0 0) 0x7f4034136960 con 0x7f40341430c0 2024-03-25T18:23:14.805 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:14.804+0000 7f402a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:23:14.805 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:14.804+0000 7f402a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:14.806 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:14.804+0000 7f402a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:14.806 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:14.804+0000 7f402a7fc640 10 monclient(hunting): my global_id is 4532 2024-03-25T18:23:14.806 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:14.804+0000 7f402a7fc640 1 -- 172.21.15.97:0/2661920998 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4018004eb0 con 0x7f40341430c0 2024-03-25T18:23:14.806 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:14.805+0000 7f402a7fc640 10 client.?.objecter ms_handle_connect 0x7f40340a98f0 2024-03-25T18:23:14.806 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:14.805+0000 7f402a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:14.813 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:14.811+0000 7f402a7fc640 1 -- 172.21.15.97:0/2661920998 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 964001336 0 0) 0x7f4034138c70 con 0x7f403413f710 2024-03-25T18:23:14.813 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:14.811+0000 7f402a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:23:14.813 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:14.811+0000 7f402a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:14.813 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:14.811+0000 7f402a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:14.813 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:14.811+0000 7f402a7fc640 10 monclient(hunting): my global_id is 4533 2024-03-25T18:23:14.813 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:14.812+0000 7f402a7fc640 1 -- 172.21.15.97:0/2661920998 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4034136960 con 0x7f403413f710 2024-03-25T18:23:14.820 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:14.819+0000 7f402a7fc640 1 -- 172.21.15.97:0/2661920998 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3836225644 0 0) 0x7f40341367d0 con 0x7f40340a98f0 2024-03-25T18:23:14.820 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:14.819+0000 7f402a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:23:14.820 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:14.819+0000 7f402a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:14.820 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:14.819+0000 7f402a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:14.820 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:14.819+0000 7f402a7fc640 10 monclient(hunting): my global_id is 4540 2024-03-25T18:23:14.820 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:14.819+0000 7f402a7fc640 1 -- 172.21.15.97:0/2661920998 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4034138c70 con 0x7f40340a98f0 2024-03-25T18:23:14.826 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:14.824+0000 7f402a7fc640 1 -- 172.21.15.97:0/2661920998 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 995309347 0 0) 0x7f4034136960 con 0x7f403413f710 2024-03-25T18:23:14.826 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:14.824+0000 7f402a7fc640 1 -- 172.21.15.97:0/2661920998 --> v1:172.21.15.97:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f40341367d0 con 0x7f403413f710 2024-03-25T18:23:14.826 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:14.824+0000 7f402a7fc640 1 -- 172.21.15.97:0/2661920998 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f40240018b0 con 0x7f403413f710 2024-03-25T18:23:14.826 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:14.824+0000 7f402a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:14.827 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:14.824+0000 7f402a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:23:14.827 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:14.824+0000 7f402a7fc640 10 monclient(hunting): dump: 2024-03-25T18:23:14.827 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:23:14.827 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:23:14.827 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:23:14.827 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:23:14.827 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:14.827 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:23:14.827 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:23:14.827 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:23:14.827 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:23:14.827 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:23:14.827 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:14.824+0000 7f402a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:23:14.827 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:14.824+0000 7f402a7fc640 1 -- 172.21.15.97:0/2661920998 <== mon.2 v1:172.21.15.97:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2999315816 0 0) 0x7f4018004eb0 con 0x7f40341430c0 2024-03-25T18:23:14.827 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:14.824+0000 7f402a7fc640 1 -- 172.21.15.97:0/2661920998 --> v1:172.21.15.97:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4034136960 con 0x7f40341430c0 2024-03-25T18:23:14.828 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:14.824+0000 7f402a7fc640 1 -- 172.21.15.97:0/2661920998 <== mon.2 v1:172.21.15.97:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f402c001b10 con 0x7f40341430c0 2024-03-25T18:23:14.828 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:14.824+0000 7f402a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:14.828 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:14.824+0000 7f402a7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:23:14.828 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:14.824+0000 7f402a7fc640 10 monclient(hunting): dump: 2024-03-25T18:23:14.828 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:23:14.828 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:23:14.828 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:23:14.828 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:23:14.828 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:14.828 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:23:14.828 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:23:14.828 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:23:14.828 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:23:14.828 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:23:14.835 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:14.833+0000 7f402a7fc640 1 -- 172.21.15.97:0/2661920998 <== mon.2 v1:172.21.15.97:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 799937628 0 0) 0x7f4034136960 con 0x7f40341430c0 2024-03-25T18:23:14.835 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:14.833+0000 7f402a7fc640 1 -- 172.21.15.97:0/2661920998 >> v1:172.21.15.98:6789/0 conn(0x7f40340a98f0 legacy=0x7f4034137610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:14.835 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:14.833+0000 7f402a7fc640 1 -- 172.21.15.97:0/2661920998 >> v1:172.21.15.97:6789/0 conn(0x7f403413f710 legacy=0x7f4034132f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:14.835 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:14.833+0000 7f402a7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:23:14.835 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:14.833+0000 7f402a7fc640 1 monclient: found mon.c 2024-03-25T18:23:14.835 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:14.833+0000 7f402a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:23:14.835 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:14.833+0000 7f402a7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.97:6790/0 2024-03-25T18:23:14.835 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:14.833+0000 7f402a7fc640 1 -- 172.21.15.97:0/2661920998 --> v1:172.21.15.97:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4034139630 con 0x7f40341430c0 2024-03-25T18:23:14.835 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:14.833+0000 7f402a7fc640 10 monclient: _finish_auth 0 2024-03-25T18:23:14.835 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:14.833+0000 7f402a7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:23:14.835 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:14.833+0000 7f402a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:23:14.836 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:14.833+0000 7f403bdc6640 5 monclient: authenticate success, global_id 4532 2024-03-25T18:23:14.836 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:14.833+0000 7f403bdc6640 10 monclient: _renew_subs 2024-03-25T18:23:14.836 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:14.833+0000 7f403bdc6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.97:6790/0 2024-03-25T18:23:14.836 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:14.833+0000 7f403bdc6640 1 -- 172.21.15.97:0/2661920998 --> v1:172.21.15.97:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4034139880 con 0x7f40341430c0 2024-03-25T18:23:14.836 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:14.833+0000 7f403bdc6640 10 client.4532.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:23:14.836 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:14.833+0000 7f403bdc6640 10 monclient: _renew_subs 2024-03-25T18:23:14.836 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:14.833+0000 7f403bdc6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.97:6790/0 2024-03-25T18:23:14.836 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:14.833+0000 7f403bdc6640 1 -- 172.21.15.97:0/2661920998 --> v1:172.21.15.97:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f40341337f0 con 0x7f40341430c0 2024-03-25T18:23:14.836 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:14.833+0000 7f403bdc6640 1 librados: init done 2024-03-25T18:23:14.837 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:14.834+0000 7f403bdc6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:14.837 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:14.834+0000 7f403bdc6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:14.837 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:14.834+0000 7f403bdc6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.97:6790/0 2024-03-25T18:23:14.837 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:14.834+0000 7f403bdc6640 1 -- 172.21.15.97:0/2661920998 --> v1:172.21.15.97:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40340627b0 con 0x7f40341430c0 2024-03-25T18:23:15.139 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:15.137+0000 7f402a7fc640 1 -- 172.21.15.97:0/2661920998 <== mon.2 v1:172.21.15.97:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f402c002000 con 0x7f40341430c0 2024-03-25T18:23:15.139 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:15.137+0000 7f402a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:23:15.139 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:15.137+0000 7f402a7fc640 1 -- 172.21.15.97:0/2661920998 <== mon.2 v1:172.21.15.97:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f402c004d00 con 0x7f40341430c0 2024-03-25T18:23:15.139 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:15.137+0000 7f402a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:15.139 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:15.137+0000 7f402a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:23:15.139 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:15.137+0000 7f402a7fc640 10 monclient: dump: 2024-03-25T18:23:15.139 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:23:15.139 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:23:15.140 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:23:15.140 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:23:15.140 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:15.140 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:23:15.140 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:23:15.140 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:23:15.140 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:23:15.140 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:23:15.140 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:15.137+0000 7f402bfff640 4 set_mon_vals no callback set 2024-03-25T18:23:15.140 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:15.138+0000 7f402a7fc640 1 -- 172.21.15.97:0/2661920998 <== mon.2 v1:172.21.15.97:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2377622137 0 0) 0x7f402c004f20 con 0x7f40341430c0 2024-03-25T18:23:15.141 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:15.138+0000 7f4039b62640 10 monclient: get_auth_request con 0x7f401806dfb0 auth_method 0 2024-03-25T18:23:15.141 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:15.139+0000 7f402a7fc640 10 client.4532.objecter ms_handle_connect 0x7f401806dfb0 2024-03-25T18:23:15.146 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:15.145+0000 7f402a7fc640 1 -- 172.21.15.97:0/2661920998 <== mon.2 v1:172.21.15.97:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 430711918 0 0) 0x7f402c054a70 con 0x7f40341430c0 2024-03-25T18:23:15.146 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:15.145+0000 7f402a7fc640 10 client.4532.objecter ms_dispatch 0x7f4034000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:23:15.146 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:15.145+0000 7f402a7fc640 3 client.4532.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:23:15.147 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:15.145+0000 7f402a7fc640 3 client.4532.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:23:15.147 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:15.145+0000 7f402a7fc640 20 client.4532.objecter dump_active .. 0 homeless 2024-03-25T18:23:15.147 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:15.145+0000 7f402a7fc640 1 -- 172.21.15.97:0/2661920998 <== mon.2 v1:172.21.15.97:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f402c050580 con 0x7f40341430c0 2024-03-25T18:23:15.147 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:15.145+0000 7f402a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:15.147 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:15.145+0000 7f402a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:23:15.399 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:15.397+0000 7f403bdc6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:15.399 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:15.397+0000 7f403bdc6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:15.399 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:15.397+0000 7f403bdc6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.97:6790/0 2024-03-25T18:23:15.399 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:15.397+0000 7f403bdc6640 1 -- 172.21.15.97:0/2661920998 --> v1:172.21.15.97:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4034074cc0 con 0x7f40341430c0 2024-03-25T18:23:15.421 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:15.419+0000 7f402a7fc640 1 -- 172.21.15.97:0/2661920998 <== mon.2 v1:172.21.15.97:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2169427412) 0x7f402c0524b0 con 0x7f40341430c0 2024-03-25T18:23:15.421 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:15.419+0000 7f402a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:15.421 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:15.419+0000 7f402a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:23:15.421 INFO:teuthology.misc.health.smithi097.stdout: 2024-03-25T18:23:15.421 INFO:teuthology.misc.health.smithi097.stdout:{"epoch":10,"fsid":"4c3d472c-79b8-486b-9e3d-2738d4306754","created":"2024-03-25T18:16:14.737309+0000","modified":"2024-03-25T18:16:20.357492+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:20.357492+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cb5152b9-262f-45c3-94b8-18d0e81df558","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6daf8793-0838-473d-a7d6-9a0d2894c75d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"136f1493-3b9c-4f43-a2e7-641482d69dae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"397d538c-4f64-4daf-9e8b-662d09f91327","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4f1df08-663f-4bda-93f8-3878ab7c24b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c4b18d70-a3ce-4b15-9b70-eae5525bd761","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8dd4033a-de0e-4f8a-96dc-59f3c1f825b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0077722e-ffa6-4a3d-9891-929d9869b286","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:23:15.424 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:15.422+0000 7f403bdc6640 10 librados: watch_flush enter 2024-03-25T18:23:15.424 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:15.422+0000 7f403bdc6640 10 librados: watch_flush exit 2024-03-25T18:23:15.424 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:15.422+0000 7f403bdc6640 20 client.4532.objecter shutdown clearing up homeless session... 2024-03-25T18:23:15.424 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:15.422+0000 7f403bdc6640 10 client.4532.objecter successfully canceled tick 2024-03-25T18:23:15.424 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:15.422+0000 7f403bdc6640 1 -- 172.21.15.97:0/2661920998 >> v1:172.21.15.98:6800/800387573 conn(0x7f401806dfb0 legacy=0x7f4018070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:15.424 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:15.422+0000 7f403bdc6640 10 monclient: shutdown 2024-03-25T18:23:15.424 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:15.422+0000 7f403bdc6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:23:15.424 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:15.422+0000 7f403bdc6640 1 -- 172.21.15.97:0/2661920998 >> v1:172.21.15.97:6790/0 conn(0x7f40341430c0 legacy=0x7f40341e2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:15.424 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:15.422+0000 7f403bdc6640 1 -- 172.21.15.97:0/2661920998 shutdown_connections 2024-03-25T18:23:15.424 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:15.422+0000 7f403bdc6640 1 -- 172.21.15.97:0/2661920998 >> 172.21.15.97:0/2661920998 conn(0x7f40340b7ea0 msgr2=0x7f4034074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:23:15.424 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:15.422+0000 7f403bdc6640 1 -- 172.21.15.97:0/2661920998 shutdown_connections 2024-03-25T18:23:15.424 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:15.422+0000 7f403bdc6640 1 -- 172.21.15.97:0/2661920998 wait complete. 2024-03-25T18:23:15.424 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:15.422+0000 7f403bdc6640 1 librados: shutdown 2024-03-25T18:23:15.436 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:23:22.242 DEBUG:teuthology.orchestra.run.smithi097:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:23:22.358 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.358+0000 7fe59b982640 10 monclient: get_monmap_and_config 2024-03-25T18:23:22.359 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.358+0000 7fe59b982640 10 monclient: build_initial_monmap 2024-03-25T18:23:22.359 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.358+0000 7fe59b982640 10 monclient: monmap: 2024-03-25T18:23:22.359 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:23:22.359 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:23:22.359 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:23:22.358752+0000 2024-03-25T18:23:22.359 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:23:22.358752+0000 2024-03-25T18:23:22.359 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:23:22.359 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:23:22.359 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:23:22.359 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T18:23:22.359 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-b 2024-03-25T18:23:22.359 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:23:22.360 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.359+0000 7fe59b982640 1 Processor -- start 2024-03-25T18:23:22.360 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.359+0000 7fe59b982640 1 -- start start 2024-03-25T18:23:22.361 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.359+0000 7fe59b982640 10 monclient: init 2024-03-25T18:23:22.361 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.359+0000 7fe59b982640 10 monclient: _reopen_session rank -1 2024-03-25T18:23:22.361 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.359+0000 7fe59b982640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:23:22.361 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.360+0000 7fe59b982640 10 monclient(hunting): picked mon.noname-b con 0x7fe5940b4d60 addr v1:172.21.15.97:6790/0 2024-03-25T18:23:22.361 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.360+0000 7fe59b982640 10 monclient(hunting): picked mon.noname-c con 0x7fe5940b1730 addr v1:172.21.15.98:6789/0 2024-03-25T18:23:22.361 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.360+0000 7fe59b982640 10 monclient(hunting): picked mon.noname-a con 0x7fe59414be90 addr v1:172.21.15.97:6789/0 2024-03-25T18:23:22.361 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.360+0000 7fe59b982640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5940af940 con 0x7fe59414be90 2024-03-25T18:23:22.361 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.360+0000 7fe59b982640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5940b0a70 con 0x7fe5940b1730 2024-03-25T18:23:22.361 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.360+0000 7fe59b982640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5940b0d50 con 0x7fe5940b4d60 2024-03-25T18:23:22.361 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.360+0000 7fe59b982640 10 monclient(hunting): _renew_subs 2024-03-25T18:23:22.361 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.360+0000 7fe59b982640 10 monclient(hunting): authenticate will time out at 8014.634967s 2024-03-25T18:23:22.367 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.367+0000 7fe598f1d640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fe5940b1730 0x7fe5940b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.97:34590/0 (socket says 172.21.15.97:34590) 2024-03-25T18:23:22.367 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.367+0000 7fe598f1d640 1 -- 172.21.15.97:0/979522117 learned_addr learned my addr 172.21.15.97:0/979522117 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:23:22.380 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.379+0000 7fe58bfff640 1 -- 172.21.15.97:0/979522117 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4203056433 0 0) 0x7fe5940b0a70 con 0x7fe5940b1730 2024-03-25T18:23:22.380 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.379+0000 7fe58bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:23:22.380 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.379+0000 7fe58bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:22.380 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.379+0000 7fe58bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:22.380 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.379+0000 7fe58bfff640 10 monclient(hunting): my global_id is 4543 2024-03-25T18:23:22.380 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.379+0000 7fe58bfff640 1 -- 172.21.15.97:0/979522117 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe580003520 con 0x7fe5940b1730 2024-03-25T18:23:22.381 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.380+0000 7fe58bfff640 1 -- 172.21.15.97:0/979522117 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3838717357 0 0) 0x7fe5940af940 con 0x7fe59414be90 2024-03-25T18:23:22.381 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.380+0000 7fe58bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:23:22.381 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.380+0000 7fe58bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:22.381 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.380+0000 7fe58bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:22.381 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.380+0000 7fe58bfff640 10 monclient(hunting): my global_id is 4536 2024-03-25T18:23:22.381 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.380+0000 7fe58bfff640 1 -- 172.21.15.97:0/979522117 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5940b0a70 con 0x7fe59414be90 2024-03-25T18:23:22.385 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.384+0000 7fe58bfff640 1 -- 172.21.15.97:0/979522117 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1658617729 0 0) 0x7fe580003520 con 0x7fe5940b1730 2024-03-25T18:23:22.385 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.384+0000 7fe58bfff640 1 -- 172.21.15.97:0/979522117 >> v1:172.21.15.97:6790/0 conn(0x7fe5940b4d60 legacy=0x7fe59413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:23:22.385 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.384+0000 7fe58bfff640 1 -- 172.21.15.97:0/979522117 >> v1:172.21.15.97:6789/0 conn(0x7fe59414be90 legacy=0x7fe59414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:22.385 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.384+0000 7fe58bfff640 10 monclient: _finish_hunting 0 2024-03-25T18:23:22.385 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.384+0000 7fe58bfff640 1 monclient: found mon.noname-c 2024-03-25T18:23:22.386 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.384+0000 7fe58bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:23:22.386 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.384+0000 7fe58bfff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.98:6789/0 2024-03-25T18:23:22.386 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.384+0000 7fe58bfff640 1 -- 172.21.15.97:0/979522117 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5940b1030 con 0x7fe5940b1730 2024-03-25T18:23:22.386 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.384+0000 7fe58bfff640 10 monclient: _finish_auth 0 2024-03-25T18:23:22.386 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.384+0000 7fe58bfff640 10 monclient: _check_auth_tickets 2024-03-25T18:23:22.386 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.384+0000 7fe58bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:23:22.386 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.384+0000 7fe58bfff640 1 -- 172.21.15.97:0/979522117 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7fe584001690 con 0x7fe5940b1730 2024-03-25T18:23:22.386 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.384+0000 7fe58bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:22.386 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.384+0000 7fe58bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:23:22.386 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.385+0000 7fe58bfff640 10 monclient: dump: 2024-03-25T18:23:22.386 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:23:22.386 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:23:22.386 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:23:22.386 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:23:22.386 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:22.387 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:23:22.387 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:23:22.387 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:23:22.387 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:23:22.387 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:23:22.387 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.385+0000 7fe59b982640 5 monclient: authenticate success, global_id 4543 2024-03-25T18:23:22.387 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.385+0000 7fe59b982640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:23:22.392 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.392+0000 7fe58bfff640 1 -- 172.21.15.97:0/979522117 <== mon.1 v1:172.21.15.98:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe5840017f0 con 0x7fe5940b1730 2024-03-25T18:23:22.393 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.392+0000 7fe58bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:23:22.393 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.392+0000 7fe58bfff640 1 -- 172.21.15.97:0/979522117 <== mon.1 v1:172.21.15.98:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7fe584001c00 con 0x7fe5940b1730 2024-03-25T18:23:22.393 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.392+0000 7fe58bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:22.393 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.392+0000 7fe58bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:23:22.393 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.392+0000 7fe58bfff640 10 monclient: dump: 2024-03-25T18:23:22.393 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:23:22.393 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:23:22.393 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:23:22.393 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:23:22.393 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:22.393 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:23:22.393 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:23:22.393 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:23:22.393 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:23:22.394 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:23:22.394 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.392+0000 7fe59b982640 10 monclient: get_monmap_and_config success 2024-03-25T18:23:22.394 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.392+0000 7fe59b982640 4 set_mon_vals no callback set 2024-03-25T18:23:22.395 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.393+0000 7fe59b982640 10 monclient: shutdown 2024-03-25T18:23:22.395 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.393+0000 7fe59b982640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:23:22.395 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.393+0000 7fe59b982640 1 -- 172.21.15.97:0/979522117 >> v1:172.21.15.98:6789/0 conn(0x7fe5940b1730 legacy=0x7fe5940b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:22.395 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.393+0000 7fe59b982640 1 -- 172.21.15.97:0/979522117 shutdown_connections 2024-03-25T18:23:22.395 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.393+0000 7fe59b982640 1 -- 172.21.15.97:0/979522117 >> 172.21.15.97:0/979522117 conn(0x7fe5940617a0 msgr2=0x7fe594061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:23:22.395 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.393+0000 7fe59b982640 1 -- 172.21.15.97:0/979522117 shutdown_connections 2024-03-25T18:23:22.395 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.393+0000 7fe59b982640 1 -- 172.21.15.97:0/979522117 wait complete. 2024-03-25T18:23:22.396 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.394+0000 7fe59b982640 10 monclient: build_initial_monmap 2024-03-25T18:23:22.396 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.394+0000 7fe59b982640 10 monclient: monmap: 2024-03-25T18:23:22.396 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:23:22.396 INFO:teuthology.misc.health.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:23:22.396 INFO:teuthology.misc.health.smithi097.stderr:last_changed 0.000000 2024-03-25T18:23:22.396 INFO:teuthology.misc.health.smithi097.stderr:created 0.000000 2024-03-25T18:23:22.396 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:23:22.396 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 1 2024-03-25T18:23:22.396 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:23:22.396 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T18:23:22.396 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-c 2024-03-25T18:23:22.396 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:23:22.396 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.394+0000 7fe59b982640 1 librados: starting msgr at 2024-03-25T18:23:22.396 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.394+0000 7fe59b982640 1 librados: starting objecter 2024-03-25T18:23:22.396 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.394+0000 7fe59b982640 1 Processor -- start 2024-03-25T18:23:22.396 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.395+0000 7fe59b982640 1 -- start start 2024-03-25T18:23:22.396 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.395+0000 7fe59b982640 1 librados: setting wanted keys 2024-03-25T18:23:22.397 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.395+0000 7fe59b982640 1 librados: calling monclient init 2024-03-25T18:23:22.397 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.395+0000 7fe59b982640 10 monclient: init 2024-03-25T18:23:22.397 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.395+0000 7fe59b982640 10 monclient: _reopen_session rank -1 2024-03-25T18:23:22.397 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.395+0000 7fe59b982640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:23:22.398 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.395+0000 7fe59b982640 10 monclient(hunting): picked mon.noname-c con 0x7fe5940b1730 addr v1:172.21.15.97:6790/0 2024-03-25T18:23:22.398 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.395+0000 7fe59b982640 10 monclient(hunting): picked mon.noname-b con 0x7fe5940b4d60 addr v1:172.21.15.98:6789/0 2024-03-25T18:23:22.398 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.395+0000 7fe59b982640 10 monclient(hunting): picked mon.noname-a con 0x7fe59414be90 addr v1:172.21.15.97:6789/0 2024-03-25T18:23:22.398 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.395+0000 7fe59b982640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe594138840 con 0x7fe59414be90 2024-03-25T18:23:22.398 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.395+0000 7fe59b982640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5941d8620 con 0x7fe5940b4d60 2024-03-25T18:23:22.398 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.395+0000 7fe59b982640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5941d79f0 con 0x7fe5940b1730 2024-03-25T18:23:22.398 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.395+0000 7fe59b982640 10 monclient(hunting): _renew_subs 2024-03-25T18:23:22.398 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.395+0000 7fe59b982640 10 monclient(hunting): authenticate will time out at 8014.670525s 2024-03-25T18:23:22.400 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.399+0000 7fe59971e640 1 --1- >> v1:172.21.15.97:6790/0 conn(0x7fe5940b1730 0x7fe5941d7f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.97:6790/0 says I am v1:172.21.15.97:53882/0 (socket says 172.21.15.97:53882) 2024-03-25T18:23:22.400 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.399+0000 7fe59971e640 1 -- 172.21.15.97:0/1417141391 learned_addr learned my addr 172.21.15.97:0/1417141391 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:23:22.400 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.400+0000 7fe589ffb640 10 client.?.objecter ms_handle_connect 0x7fe5940b1730 2024-03-25T18:23:22.400 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.400+0000 7fe589ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:22.411 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.410+0000 7fe589ffb640 1 -- 172.21.15.97:0/1417141391 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1847413980 0 0) 0x7fe5941d79f0 con 0x7fe5940b1730 2024-03-25T18:23:22.411 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.410+0000 7fe589ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:23:22.411 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.410+0000 7fe589ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:22.411 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.410+0000 7fe589ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:22.411 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.410+0000 7fe589ffb640 10 monclient(hunting): my global_id is 4535 2024-03-25T18:23:22.411 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.410+0000 7fe589ffb640 1 -- 172.21.15.97:0/1417141391 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe578004eb0 con 0x7fe5940b1730 2024-03-25T18:23:22.415 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.415+0000 7fe589ffb640 10 client.?.objecter ms_handle_connect 0x7fe59414be90 2024-03-25T18:23:22.415 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.415+0000 7fe589ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:22.433 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.433+0000 7fe589ffb640 1 -- 172.21.15.97:0/1417141391 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2521671466 0 0) 0x7fe594138840 con 0x7fe59414be90 2024-03-25T18:23:22.434 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.433+0000 7fe589ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:23:22.434 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.433+0000 7fe589ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:22.434 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.433+0000 7fe589ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:22.434 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.433+0000 7fe589ffb640 10 monclient(hunting): my global_id is 4539 2024-03-25T18:23:22.434 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.433+0000 7fe589ffb640 1 -- 172.21.15.97:0/1417141391 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5941d79f0 con 0x7fe59414be90 2024-03-25T18:23:22.436 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.435+0000 7fe589ffb640 10 client.?.objecter ms_handle_connect 0x7fe5940b4d60 2024-03-25T18:23:22.436 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.435+0000 7fe589ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:22.442 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.441+0000 7fe589ffb640 1 -- 172.21.15.97:0/1417141391 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 545731477 0 0) 0x7fe5941d79f0 con 0x7fe59414be90 2024-03-25T18:23:22.442 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.441+0000 7fe589ffb640 1 -- 172.21.15.97:0/1417141391 --> v1:172.21.15.97:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe594138840 con 0x7fe59414be90 2024-03-25T18:23:22.442 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.441+0000 7fe589ffb640 1 -- 172.21.15.97:0/1417141391 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7fe58c001b50 con 0x7fe59414be90 2024-03-25T18:23:22.442 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.441+0000 7fe589ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:22.442 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.441+0000 7fe589ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:23:22.442 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.441+0000 7fe589ffb640 10 monclient(hunting): dump: 2024-03-25T18:23:22.442 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:23:22.442 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:23:22.443 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:23:22.443 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:23:22.443 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:22.443 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:23:22.443 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:23:22.443 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:23:22.443 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:23:22.443 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:23:22.443 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.441+0000 7fe589ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:23:22.444 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.442+0000 7fe589ffb640 1 -- 172.21.15.97:0/1417141391 <== mon.2 v1:172.21.15.97:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1747825037 0 0) 0x7fe578004eb0 con 0x7fe5940b1730 2024-03-25T18:23:22.444 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.442+0000 7fe589ffb640 1 -- 172.21.15.97:0/1417141391 --> v1:172.21.15.97:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe5941d79f0 con 0x7fe5940b1730 2024-03-25T18:23:22.444 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.442+0000 7fe589ffb640 1 -- 172.21.15.97:0/1417141391 <== mon.2 v1:172.21.15.97:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7fe590001960 con 0x7fe5940b1730 2024-03-25T18:23:22.444 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.443+0000 7fe589ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:22.444 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.443+0000 7fe589ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:23:22.444 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.443+0000 7fe589ffb640 10 monclient(hunting): dump: 2024-03-25T18:23:22.444 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:23:22.444 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:23:22.444 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:23:22.445 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:23:22.445 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:22.445 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:23:22.445 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:23:22.445 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:23:22.445 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:23:22.445 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:23:22.454 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.454+0000 7fe589ffb640 1 -- 172.21.15.97:0/1417141391 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 416554299 0 0) 0x7fe5941d8620 con 0x7fe5940b4d60 2024-03-25T18:23:22.454 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.454+0000 7fe589ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:23:22.454 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.454+0000 7fe589ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:22.454 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.454+0000 7fe589ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:22.454 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.454+0000 7fe589ffb640 10 monclient(hunting): my global_id is 4546 2024-03-25T18:23:22.454 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.454+0000 7fe589ffb640 1 -- 172.21.15.97:0/1417141391 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe578004eb0 con 0x7fe5940b4d60 2024-03-25T18:23:22.459 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.458+0000 7fe589ffb640 1 -- 172.21.15.97:0/1417141391 <== mon.0 v1:172.21.15.97:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 926789514 0 0) 0x7fe594138840 con 0x7fe59414be90 2024-03-25T18:23:22.459 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.458+0000 7fe589ffb640 1 -- 172.21.15.97:0/1417141391 >> v1:172.21.15.97:6790/0 conn(0x7fe5940b1730 legacy=0x7fe5941d7f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:22.459 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.458+0000 7fe589ffb640 1 -- 172.21.15.97:0/1417141391 >> v1:172.21.15.98:6789/0 conn(0x7fe5940b4d60 legacy=0x7fe594137a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:22.459 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.458+0000 7fe589ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:23:22.460 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.458+0000 7fe589ffb640 1 monclient: found mon.a 2024-03-25T18:23:22.460 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.458+0000 7fe589ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:23:22.460 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.458+0000 7fe589ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:23:22.460 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.458+0000 7fe589ffb640 1 -- 172.21.15.97:0/1417141391 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe594142f20 con 0x7fe59414be90 2024-03-25T18:23:22.460 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.458+0000 7fe589ffb640 10 monclient: _finish_auth 0 2024-03-25T18:23:22.460 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.458+0000 7fe589ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:23:22.460 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.458+0000 7fe589ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:23:22.460 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.458+0000 7fe59b982640 5 monclient: authenticate success, global_id 4539 2024-03-25T18:23:22.460 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.458+0000 7fe59b982640 10 monclient: _renew_subs 2024-03-25T18:23:22.460 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.458+0000 7fe59b982640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:23:22.460 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.458+0000 7fe59b982640 1 -- 172.21.15.97:0/1417141391 --> v1:172.21.15.97:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe594143170 con 0x7fe59414be90 2024-03-25T18:23:22.460 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.458+0000 7fe59b982640 10 client.4539.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:23:22.460 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.458+0000 7fe59b982640 10 monclient: _renew_subs 2024-03-25T18:23:22.461 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.458+0000 7fe59b982640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:23:22.461 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.458+0000 7fe59b982640 1 -- 172.21.15.97:0/1417141391 --> v1:172.21.15.97:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe594140560 con 0x7fe59414be90 2024-03-25T18:23:22.461 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.458+0000 7fe59b982640 1 librados: init done 2024-03-25T18:23:22.461 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.459+0000 7fe59b982640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:22.461 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.459+0000 7fe59b982640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:22.462 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.459+0000 7fe59b982640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:23:22.462 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.459+0000 7fe59b982640 1 -- 172.21.15.97:0/1417141391 --> v1:172.21.15.97:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe594062100 con 0x7fe59414be90 2024-03-25T18:23:22.772 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.771+0000 7fe589ffb640 1 -- 172.21.15.97:0/1417141391 <== mon.0 v1:172.21.15.97:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe58c002040 con 0x7fe59414be90 2024-03-25T18:23:22.772 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.771+0000 7fe589ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:23:22.772 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.771+0000 7fe589ffb640 1 -- 172.21.15.97:0/1417141391 <== mon.0 v1:172.21.15.97:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7fe58c003b30 con 0x7fe59414be90 2024-03-25T18:23:22.772 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.771+0000 7fe589ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:22.772 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.771+0000 7fe589ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:23:22.772 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.771+0000 7fe589ffb640 10 monclient: dump: 2024-03-25T18:23:22.772 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:23:22.772 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:23:22.772 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:23:22.772 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:23:22.773 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:22.773 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:23:22.773 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:23:22.773 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:23:22.773 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:23:22.773 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:23:22.773 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.771+0000 7fe58b7fe640 4 set_mon_vals no callback set 2024-03-25T18:23:22.773 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.772+0000 7fe589ffb640 1 -- 172.21.15.97:0/1417141391 <== mon.0 v1:172.21.15.97:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2377622137 0 0) 0x7fe58c019b00 con 0x7fe59414be90 2024-03-25T18:23:22.774 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.773+0000 7fe59971e640 10 monclient: get_auth_request con 0x7fe57806df00 auth_method 0 2024-03-25T18:23:22.774 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.773+0000 7fe589ffb640 10 client.4539.objecter ms_handle_connect 0x7fe57806df00 2024-03-25T18:23:22.779 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.778+0000 7fe589ffb640 1 -- 172.21.15.97:0/1417141391 <== mon.0 v1:172.21.15.97:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 430711918 0 0) 0x7fe58c053790 con 0x7fe59414be90 2024-03-25T18:23:22.779 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.778+0000 7fe589ffb640 10 client.4539.objecter ms_dispatch 0x7fe594000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:23:22.779 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.778+0000 7fe589ffb640 3 client.4539.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:23:22.779 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.778+0000 7fe589ffb640 3 client.4539.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:23:22.779 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.778+0000 7fe589ffb640 20 client.4539.objecter dump_active .. 0 homeless 2024-03-25T18:23:22.779 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.778+0000 7fe589ffb640 1 -- 172.21.15.97:0/1417141391 <== mon.0 v1:172.21.15.97:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe58c04f2a0 con 0x7fe59414be90 2024-03-25T18:23:22.779 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.778+0000 7fe589ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:22.779 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:22.778+0000 7fe589ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:23:23.030 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:23.030+0000 7fe59b982640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:23.030 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:23.030+0000 7fe59b982640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:23.030 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:23.030+0000 7fe59b982640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:23:23.030 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:23.030+0000 7fe59b982640 1 -- 172.21.15.97:0/1417141391 --> v1:172.21.15.97:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe5941d7b80 con 0x7fe59414be90 2024-03-25T18:23:23.062 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:23.062+0000 7fe589ffb640 1 -- 172.21.15.97:0/1417141391 <== mon.0 v1:172.21.15.97:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2169427412) 0x7fe58c0511d0 con 0x7fe59414be90 2024-03-25T18:23:23.062 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:23.062+0000 7fe589ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:23.062 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:23.062+0000 7fe589ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:23:23.063 INFO:teuthology.misc.health.smithi097.stdout: 2024-03-25T18:23:23.063 INFO:teuthology.misc.health.smithi097.stdout:{"epoch":10,"fsid":"4c3d472c-79b8-486b-9e3d-2738d4306754","created":"2024-03-25T18:16:14.737309+0000","modified":"2024-03-25T18:16:20.357492+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:20.357492+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cb5152b9-262f-45c3-94b8-18d0e81df558","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6daf8793-0838-473d-a7d6-9a0d2894c75d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"136f1493-3b9c-4f43-a2e7-641482d69dae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"397d538c-4f64-4daf-9e8b-662d09f91327","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4f1df08-663f-4bda-93f8-3878ab7c24b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c4b18d70-a3ce-4b15-9b70-eae5525bd761","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8dd4033a-de0e-4f8a-96dc-59f3c1f825b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0077722e-ffa6-4a3d-9891-929d9869b286","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:23:23.065 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:23.064+0000 7fe59b982640 10 librados: watch_flush enter 2024-03-25T18:23:23.065 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:23.064+0000 7fe59b982640 10 librados: watch_flush exit 2024-03-25T18:23:23.065 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:23.064+0000 7fe59b982640 20 client.4539.objecter shutdown clearing up homeless session... 2024-03-25T18:23:23.065 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:23.064+0000 7fe59b982640 10 client.4539.objecter successfully canceled tick 2024-03-25T18:23:23.065 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:23.064+0000 7fe59b982640 1 -- 172.21.15.97:0/1417141391 >> v1:172.21.15.98:6800/800387573 conn(0x7fe57806df00 legacy=0x7fe5780703c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:23.065 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:23.064+0000 7fe59b982640 10 monclient: shutdown 2024-03-25T18:23:23.065 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:23.064+0000 7fe59b982640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:23:23.065 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:23.064+0000 7fe59b982640 1 -- 172.21.15.97:0/1417141391 >> v1:172.21.15.97:6789/0 conn(0x7fe59414be90 legacy=0x7fe594138130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:23.066 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:23.065+0000 7fe59b982640 1 -- 172.21.15.97:0/1417141391 shutdown_connections 2024-03-25T18:23:23.066 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:23.065+0000 7fe59b982640 1 -- 172.21.15.97:0/1417141391 >> 172.21.15.97:0/1417141391 conn(0x7fe5940617a0 msgr2=0x7fe5940b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:23:23.066 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:23.065+0000 7fe59b982640 1 -- 172.21.15.97:0/1417141391 shutdown_connections 2024-03-25T18:23:23.066 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:23.065+0000 7fe59b982640 1 -- 172.21.15.97:0/1417141391 wait complete. 2024-03-25T18:23:23.066 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:23.065+0000 7fe59b982640 1 librados: shutdown 2024-03-25T18:23:23.078 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:23:29.884 DEBUG:teuthology.orchestra.run.smithi097:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:23:30.001 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:29.999+0000 7f2d1cac9640 10 monclient: get_monmap_and_config 2024-03-25T18:23:30.001 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:29.999+0000 7f2d1cac9640 10 monclient: build_initial_monmap 2024-03-25T18:23:30.001 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:29.999+0000 7f2d1cac9640 10 monclient: monmap: 2024-03-25T18:23:30.001 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:23:30.001 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:23:30.001 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:23:30.000775+0000 2024-03-25T18:23:30.001 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:23:30.000775+0000 2024-03-25T18:23:30.001 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:23:30.001 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:23:30.001 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:23:30.002 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T18:23:30.002 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-b 2024-03-25T18:23:30.002 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:23:30.003 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.000+0000 7f2d1cac9640 1 Processor -- start 2024-03-25T18:23:30.003 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.000+0000 7f2d1cac9640 1 -- start start 2024-03-25T18:23:30.003 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.000+0000 7f2d1cac9640 10 monclient: init 2024-03-25T18:23:30.003 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.001+0000 7f2d1cac9640 10 monclient: _reopen_session rank -1 2024-03-25T18:23:30.003 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.001+0000 7f2d1cac9640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:23:30.003 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.001+0000 7f2d1cac9640 10 monclient(hunting): picked mon.noname-c con 0x7f2d180a98f0 addr v1:172.21.15.98:6789/0 2024-03-25T18:23:30.003 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.001+0000 7f2d1cac9640 10 monclient(hunting): picked mon.noname-a con 0x7f2d1813f710 addr v1:172.21.15.97:6789/0 2024-03-25T18:23:30.003 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.001+0000 7f2d1cac9640 10 monclient(hunting): picked mon.noname-b con 0x7f2d181430c0 addr v1:172.21.15.97:6790/0 2024-03-25T18:23:30.003 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.001+0000 7f2d1cac9640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d18142520 con 0x7f2d1813f710 2024-03-25T18:23:30.003 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.001+0000 7f2d1cac9640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d180b28a0 con 0x7f2d180a98f0 2024-03-25T18:23:30.003 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.001+0000 7f2d1cac9640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d180b2a30 con 0x7f2d181430c0 2024-03-25T18:23:30.003 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.001+0000 7f2d1cac9640 10 monclient(hunting): _renew_subs 2024-03-25T18:23:30.003 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.001+0000 7f2d1cac9640 10 monclient(hunting): authenticate will time out at 8022.277571s 2024-03-25T18:23:30.012 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.011+0000 7f2d15d9b640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f2d1813f710 0x7f2d181429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:60242/0 (socket says 172.21.15.97:60242) 2024-03-25T18:23:30.012 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.011+0000 7f2d15d9b640 1 -- 172.21.15.97:0/1715310380 learned_addr learned my addr 172.21.15.97:0/1715310380 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:23:30.023 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.021+0000 7f2d1559a640 1 -- 172.21.15.97:0/1715310380 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 302908059 0 0) 0x7f2d18142520 con 0x7f2d1813f710 2024-03-25T18:23:30.023 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.021+0000 7f2d1559a640 10 monclient(hunting): _negotiate 2024-03-25T18:23:30.023 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.021+0000 7f2d1559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:30.023 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.021+0000 7f2d1559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:30.023 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.021+0000 7f2d1559a640 10 monclient(hunting): my global_id is 4542 2024-03-25T18:23:30.023 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.021+0000 7f2d1559a640 1 -- 172.21.15.97:0/1715310380 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d00003520 con 0x7f2d1813f710 2024-03-25T18:23:30.031 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.029+0000 7f2d1559a640 1 -- 172.21.15.97:0/1715310380 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3933646293 0 0) 0x7f2d00003520 con 0x7f2d1813f710 2024-03-25T18:23:30.031 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.029+0000 7f2d1559a640 1 -- 172.21.15.97:0/1715310380 >> v1:172.21.15.97:6790/0 conn(0x7f2d181430c0 legacy=0x7f2d1814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:30.031 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.029+0000 7f2d1559a640 1 -- 172.21.15.97:0/1715310380 >> v1:172.21.15.98:6789/0 conn(0x7f2d180a98f0 legacy=0x7f2d180a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:30.032 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.029+0000 7f2d1559a640 10 monclient: _finish_hunting 0 2024-03-25T18:23:30.032 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.029+0000 7f2d1559a640 1 monclient: found mon.noname-a 2024-03-25T18:23:30.032 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.029+0000 7f2d1559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:23:30.032 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.029+0000 7f2d1559a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.97:6789/0 2024-03-25T18:23:30.032 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.029+0000 7f2d1559a640 1 -- 172.21.15.97:0/1715310380 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d180b44c0 con 0x7f2d1813f710 2024-03-25T18:23:30.032 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.029+0000 7f2d1559a640 10 monclient: _finish_auth 0 2024-03-25T18:23:30.032 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.029+0000 7f2d1559a640 10 monclient: _check_auth_tickets 2024-03-25T18:23:30.032 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.030+0000 7f2d1559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:23:30.032 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.030+0000 7f2d1559a640 1 -- 172.21.15.97:0/1715310380 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f2d0c001690 con 0x7f2d1813f710 2024-03-25T18:23:30.032 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.030+0000 7f2d1cac9640 5 monclient: authenticate success, global_id 4542 2024-03-25T18:23:30.032 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.030+0000 7f2d1cac9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:23:30.032 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.030+0000 7f2d1559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:30.032 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.030+0000 7f2d1559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:23:30.032 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.030+0000 7f2d1559a640 10 monclient: dump: 2024-03-25T18:23:30.032 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:23:30.033 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:23:30.033 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:23:30.033 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:23:30.033 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:30.033 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:23:30.033 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:23:30.033 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:23:30.033 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:23:30.033 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:23:30.033 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.030+0000 7f2d1cac9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:23:30.050 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.049+0000 7f2d1559a640 1 -- 172.21.15.97:0/1715310380 <== mon.0 v1:172.21.15.97:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2d0c0017f0 con 0x7f2d1813f710 2024-03-25T18:23:30.051 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.049+0000 7f2d1559a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:23:30.051 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.049+0000 7f2d1559a640 1 -- 172.21.15.97:0/1715310380 <== mon.0 v1:172.21.15.97:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f2d0c001c00 con 0x7f2d1813f710 2024-03-25T18:23:30.051 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.049+0000 7f2d1cac9640 10 monclient: get_monmap_and_config success 2024-03-25T18:23:30.051 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.049+0000 7f2d1cac9640 4 set_mon_vals no callback set 2024-03-25T18:23:30.051 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.049+0000 7f2d1559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:30.051 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.049+0000 7f2d1559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:23:30.051 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.049+0000 7f2d1559a640 10 monclient: dump: 2024-03-25T18:23:30.051 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:23:30.051 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:23:30.051 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:23:30.051 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:23:30.051 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:30.051 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:23:30.051 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:23:30.051 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:23:30.052 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:23:30.052 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:23:30.052 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.050+0000 7f2d1cac9640 10 monclient: shutdown 2024-03-25T18:23:30.052 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.050+0000 7f2d1cac9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:23:30.052 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.050+0000 7f2d1cac9640 1 -- 172.21.15.97:0/1715310380 >> v1:172.21.15.97:6789/0 conn(0x7f2d1813f710 legacy=0x7f2d181429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:30.052 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.050+0000 7f2d1cac9640 1 -- 172.21.15.97:0/1715310380 shutdown_connections 2024-03-25T18:23:30.052 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.050+0000 7f2d1cac9640 1 -- 172.21.15.97:0/1715310380 >> 172.21.15.97:0/1715310380 conn(0x7f2d180b7ea0 msgr2=0x7f2d180b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:23:30.053 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.050+0000 7f2d1cac9640 1 -- 172.21.15.97:0/1715310380 shutdown_connections 2024-03-25T18:23:30.053 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.050+0000 7f2d1cac9640 1 -- 172.21.15.97:0/1715310380 wait complete. 2024-03-25T18:23:30.053 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.051+0000 7f2d1cac9640 10 monclient: build_initial_monmap 2024-03-25T18:23:30.053 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.051+0000 7f2d1cac9640 10 monclient: monmap: 2024-03-25T18:23:30.054 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:23:30.054 INFO:teuthology.misc.health.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:23:30.054 INFO:teuthology.misc.health.smithi097.stderr:last_changed 0.000000 2024-03-25T18:23:30.054 INFO:teuthology.misc.health.smithi097.stderr:created 0.000000 2024-03-25T18:23:30.054 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:23:30.054 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 1 2024-03-25T18:23:30.054 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:23:30.054 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T18:23:30.054 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-c 2024-03-25T18:23:30.054 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:23:30.054 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.051+0000 7f2d1cac9640 1 librados: starting msgr at 2024-03-25T18:23:30.054 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.051+0000 7f2d1cac9640 1 librados: starting objecter 2024-03-25T18:23:30.054 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.052+0000 7f2d1cac9640 1 Processor -- start 2024-03-25T18:23:30.054 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.052+0000 7f2d1cac9640 1 -- start start 2024-03-25T18:23:30.054 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.052+0000 7f2d1cac9640 1 librados: setting wanted keys 2024-03-25T18:23:30.055 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.052+0000 7f2d1cac9640 1 librados: calling monclient init 2024-03-25T18:23:30.055 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.052+0000 7f2d1cac9640 10 monclient: init 2024-03-25T18:23:30.055 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.053+0000 7f2d1cac9640 10 monclient: _reopen_session rank -1 2024-03-25T18:23:30.056 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.053+0000 7f2d1cac9640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:23:30.056 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.053+0000 7f2d1cac9640 10 monclient(hunting): picked mon.noname-b con 0x7f2d180a98f0 addr v1:172.21.15.98:6789/0 2024-03-25T18:23:30.056 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.053+0000 7f2d1cac9640 10 monclient(hunting): picked mon.noname-c con 0x7f2d1813f710 addr v1:172.21.15.97:6790/0 2024-03-25T18:23:30.056 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.053+0000 7f2d1cac9640 10 monclient(hunting): picked mon.noname-a con 0x7f2d181430c0 addr v1:172.21.15.97:6789/0 2024-03-25T18:23:30.056 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.053+0000 7f2d1cac9640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d18139a80 con 0x7f2d181430c0 2024-03-25T18:23:30.056 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.053+0000 7f2d1cac9640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d18137450 con 0x7f2d180a98f0 2024-03-25T18:23:30.056 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.053+0000 7f2d1cac9640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d181d7ba0 con 0x7f2d1813f710 2024-03-25T18:23:30.056 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.053+0000 7f2d1cac9640 10 monclient(hunting): _renew_subs 2024-03-25T18:23:30.056 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.053+0000 7f2d1cac9640 10 monclient(hunting): authenticate will time out at 8022.329193s 2024-03-25T18:23:30.061 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.060+0000 7f2d16d9d640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f2d181430c0 0x7f2d18132f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:60294/0 (socket says 172.21.15.97:60294) 2024-03-25T18:23:30.061 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.060+0000 7f2d16d9d640 1 -- 172.21.15.97:0/3840413004 learned_addr learned my addr 172.21.15.97:0/3840413004 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:23:30.064 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.062+0000 7f2d06ffd640 10 client.?.objecter ms_handle_connect 0x7f2d180a98f0 2024-03-25T18:23:30.064 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.062+0000 7f2d06ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:30.064 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.062+0000 7f2d06ffd640 10 client.?.objecter ms_handle_connect 0x7f2d181430c0 2024-03-25T18:23:30.064 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.062+0000 7f2d06ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:30.066 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.064+0000 7f2d06ffd640 10 client.?.objecter ms_handle_connect 0x7f2d1813f710 2024-03-25T18:23:30.066 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.064+0000 7f2d06ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:30.077 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.076+0000 7f2d06ffd640 1 -- 172.21.15.97:0/3840413004 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 540910988 0 0) 0x7f2d180b2a30 con 0x7f2d181430c0 2024-03-25T18:23:30.077 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.076+0000 7f2d06ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:23:30.077 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.076+0000 7f2d06ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:30.077 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.076+0000 7f2d06ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:30.077 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.076+0000 7f2d06ffd640 10 monclient(hunting): my global_id is 4545 2024-03-25T18:23:30.077 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.076+0000 7f2d06ffd640 1 -- 172.21.15.97:0/3840413004 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2cf8004eb0 con 0x7f2d181430c0 2024-03-25T18:23:30.078 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.077+0000 7f2d06ffd640 1 -- 172.21.15.97:0/3840413004 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 107812413 0 0) 0x7f2d181d7ba0 con 0x7f2d1813f710 2024-03-25T18:23:30.079 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.077+0000 7f2d06ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:23:30.079 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.077+0000 7f2d06ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:30.079 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.077+0000 7f2d06ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:30.079 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.077+0000 7f2d06ffd640 10 monclient(hunting): my global_id is 4541 2024-03-25T18:23:30.079 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.077+0000 7f2d06ffd640 1 -- 172.21.15.97:0/3840413004 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d180b2a30 con 0x7f2d1813f710 2024-03-25T18:23:30.079 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.078+0000 7f2d06ffd640 1 -- 172.21.15.97:0/3840413004 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2930119166 0 0) 0x7f2d18137450 con 0x7f2d180a98f0 2024-03-25T18:23:30.079 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.078+0000 7f2d06ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:23:30.079 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.078+0000 7f2d06ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:30.080 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.078+0000 7f2d06ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:30.080 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.078+0000 7f2d06ffd640 10 monclient(hunting): my global_id is 4552 2024-03-25T18:23:30.080 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.078+0000 7f2d06ffd640 1 -- 172.21.15.97:0/3840413004 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d181d7ba0 con 0x7f2d180a98f0 2024-03-25T18:23:30.088 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.087+0000 7f2d06ffd640 1 -- 172.21.15.97:0/3840413004 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1700333530 0 0) 0x7f2cf8004eb0 con 0x7f2d181430c0 2024-03-25T18:23:30.088 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.087+0000 7f2d06ffd640 1 -- 172.21.15.97:0/3840413004 --> v1:172.21.15.97:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2d18137450 con 0x7f2d181430c0 2024-03-25T18:23:30.088 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.087+0000 7f2d06ffd640 1 -- 172.21.15.97:0/3840413004 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f2d10001770 con 0x7f2d181430c0 2024-03-25T18:23:30.088 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.087+0000 7f2d06ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:30.088 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.087+0000 7f2d06ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:23:30.088 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.087+0000 7f2d06ffd640 10 monclient(hunting): dump: 2024-03-25T18:23:30.089 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:23:30.089 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:23:30.089 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:23:30.089 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:23:30.089 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:30.089 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:23:30.089 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:23:30.089 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:23:30.089 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:23:30.089 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:23:30.089 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.087+0000 7f2d06ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:23:30.094 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.093+0000 7f2d06ffd640 1 -- 172.21.15.97:0/3840413004 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2525771497 0 0) 0x7f2d181d7ba0 con 0x7f2d180a98f0 2024-03-25T18:23:30.095 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.093+0000 7f2d06ffd640 1 -- 172.21.15.97:0/3840413004 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2cf8004eb0 con 0x7f2d180a98f0 2024-03-25T18:23:30.095 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.093+0000 7f2d06ffd640 1 -- 172.21.15.97:0/3840413004 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f2d08001c60 con 0x7f2d180a98f0 2024-03-25T18:23:30.095 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.093+0000 7f2d06ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:30.095 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.093+0000 7f2d06ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:23:30.095 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.093+0000 7f2d06ffd640 10 monclient(hunting): dump: 2024-03-25T18:23:30.095 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:23:30.095 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:23:30.095 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:23:30.095 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:23:30.095 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:30.095 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:23:30.095 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:23:30.095 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:23:30.095 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:23:30.096 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:23:30.097 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.095+0000 7f2d06ffd640 1 -- 172.21.15.97:0/3840413004 <== mon.2 v1:172.21.15.97:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2037718692 0 0) 0x7f2d180b2a30 con 0x7f2d1813f710 2024-03-25T18:23:30.097 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.095+0000 7f2d06ffd640 1 -- 172.21.15.97:0/3840413004 --> v1:172.21.15.97:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2d181d7ba0 con 0x7f2d1813f710 2024-03-25T18:23:30.097 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.095+0000 7f2d06ffd640 1 -- 172.21.15.97:0/3840413004 <== mon.2 v1:172.21.15.97:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f2d0c001b40 con 0x7f2d1813f710 2024-03-25T18:23:30.097 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.095+0000 7f2d06ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:30.097 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.095+0000 7f2d06ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:23:30.097 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.095+0000 7f2d06ffd640 10 monclient(hunting): dump: 2024-03-25T18:23:30.097 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:23:30.097 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:23:30.097 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:23:30.097 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:23:30.097 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:30.097 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:23:30.097 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:23:30.097 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:23:30.097 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:23:30.097 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:23:30.100 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.098+0000 7f2d06ffd640 1 -- 172.21.15.97:0/3840413004 <== mon.1 v1:172.21.15.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1133752047 0 0) 0x7f2cf8004eb0 con 0x7f2d180a98f0 2024-03-25T18:23:30.100 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.098+0000 7f2d06ffd640 1 -- 172.21.15.97:0/3840413004 >> v1:172.21.15.97:6790/0 conn(0x7f2d1813f710 legacy=0x7f2d181d7490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:30.100 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.098+0000 7f2d06ffd640 1 -- 172.21.15.97:0/3840413004 >> v1:172.21.15.97:6789/0 conn(0x7f2d181430c0 legacy=0x7f2d18132f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:30.100 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.098+0000 7f2d06ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:23:30.100 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.098+0000 7f2d06ffd640 1 monclient: found mon.b 2024-03-25T18:23:30.100 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.098+0000 7f2d06ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:23:30.100 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.098+0000 7f2d06ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:23:30.100 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.098+0000 7f2d06ffd640 1 -- 172.21.15.97:0/3840413004 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d181368f0 con 0x7f2d180a98f0 2024-03-25T18:23:30.100 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.098+0000 7f2d06ffd640 10 monclient: _finish_auth 0 2024-03-25T18:23:30.100 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.098+0000 7f2d06ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:23:30.100 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.098+0000 7f2d06ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:23:30.100 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.098+0000 7f2d1cac9640 5 monclient: authenticate success, global_id 4552 2024-03-25T18:23:30.101 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.098+0000 7f2d1cac9640 10 monclient: _renew_subs 2024-03-25T18:23:30.101 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.098+0000 7f2d1cac9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:23:30.101 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.098+0000 7f2d1cac9640 1 -- 172.21.15.97:0/3840413004 --> v1:172.21.15.98:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d18136b40 con 0x7f2d180a98f0 2024-03-25T18:23:30.101 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.098+0000 7f2d1cac9640 10 client.4552.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:23:30.101 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.098+0000 7f2d1cac9640 10 monclient: _renew_subs 2024-03-25T18:23:30.101 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.098+0000 7f2d1cac9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:23:30.101 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.098+0000 7f2d1cac9640 1 -- 172.21.15.97:0/3840413004 --> v1:172.21.15.98:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d18137c00 con 0x7f2d180a98f0 2024-03-25T18:23:30.101 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.098+0000 7f2d1cac9640 1 librados: init done 2024-03-25T18:23:30.101 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.099+0000 7f2d1cac9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:30.101 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.099+0000 7f2d1cac9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:30.102 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.099+0000 7f2d1cac9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:23:30.102 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.099+0000 7f2d1cac9640 1 -- 172.21.15.97:0/3840413004 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d18074cc0 con 0x7f2d180a98f0 2024-03-25T18:23:30.381 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.379+0000 7f2d06ffd640 1 -- 172.21.15.97:0/3840413004 <== mon.1 v1:172.21.15.98:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2d08003970 con 0x7f2d180a98f0 2024-03-25T18:23:30.381 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.379+0000 7f2d06ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:23:30.381 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.379+0000 7f2d06ffd640 1 -- 172.21.15.97:0/3840413004 <== mon.1 v1:172.21.15.98:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f2d08003ae0 con 0x7f2d180a98f0 2024-03-25T18:23:30.381 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.379+0000 7f2d06ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:30.381 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.379+0000 7f2d06ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:23:30.381 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.379+0000 7f2d06ffd640 10 monclient: dump: 2024-03-25T18:23:30.381 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:23:30.381 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:23:30.382 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:23:30.382 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:23:30.382 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:30.382 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:23:30.382 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:23:30.382 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:23:30.382 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:23:30.382 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:23:30.382 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.380+0000 7f2d14d99640 4 set_mon_vals no callback set 2024-03-25T18:23:30.383 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.380+0000 7f2d06ffd640 1 -- 172.21.15.97:0/3840413004 <== mon.1 v1:172.21.15.98:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2377622137 0 0) 0x7f2d08019ab0 con 0x7f2d180a98f0 2024-03-25T18:23:30.383 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.381+0000 7f2d15d9b640 10 monclient: get_auth_request con 0x7f2cf806e110 auth_method 0 2024-03-25T18:23:30.383 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.381+0000 7f2d06ffd640 10 client.4552.objecter ms_handle_connect 0x7f2cf806e110 2024-03-25T18:23:30.389 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.388+0000 7f2d06ffd640 1 -- 172.21.15.97:0/3840413004 <== mon.1 v1:172.21.15.98:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 430711918 0 0) 0x7f2d080536d0 con 0x7f2d180a98f0 2024-03-25T18:23:30.389 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.388+0000 7f2d06ffd640 10 client.4552.objecter ms_dispatch 0x7f2d18000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:23:30.390 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.388+0000 7f2d06ffd640 3 client.4552.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:23:30.390 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.388+0000 7f2d06ffd640 3 client.4552.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:23:30.390 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.388+0000 7f2d06ffd640 20 client.4552.objecter dump_active .. 0 homeless 2024-03-25T18:23:30.390 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.389+0000 7f2d06ffd640 1 -- 172.21.15.97:0/3840413004 <== mon.1 v1:172.21.15.98:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2d0804f1e0 con 0x7f2d180a98f0 2024-03-25T18:23:30.390 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.389+0000 7f2d06ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:30.390 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.389+0000 7f2d06ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:23:30.639 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.637+0000 7f2d1cac9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:30.639 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.637+0000 7f2d1cac9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:30.639 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.637+0000 7f2d1cac9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:23:30.639 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.637+0000 7f2d1cac9640 1 -- 172.21.15.97:0/3840413004 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2d181375e0 con 0x7f2d180a98f0 2024-03-25T18:23:30.730 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.729+0000 7f2d06ffd640 1 -- 172.21.15.97:0/3840413004 <== mon.1 v1:172.21.15.98:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2169427412) 0x7f2d08051110 con 0x7f2d180a98f0 2024-03-25T18:23:30.731 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.729+0000 7f2d06ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:30.731 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.729+0000 7f2d06ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:23:30.731 INFO:teuthology.misc.health.smithi097.stdout: 2024-03-25T18:23:30.731 INFO:teuthology.misc.health.smithi097.stdout:{"epoch":10,"fsid":"4c3d472c-79b8-486b-9e3d-2738d4306754","created":"2024-03-25T18:16:14.737309+0000","modified":"2024-03-25T18:16:20.357492+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:20.357492+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cb5152b9-262f-45c3-94b8-18d0e81df558","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6daf8793-0838-473d-a7d6-9a0d2894c75d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"136f1493-3b9c-4f43-a2e7-641482d69dae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"397d538c-4f64-4daf-9e8b-662d09f91327","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4f1df08-663f-4bda-93f8-3878ab7c24b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c4b18d70-a3ce-4b15-9b70-eae5525bd761","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8dd4033a-de0e-4f8a-96dc-59f3c1f825b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0077722e-ffa6-4a3d-9891-929d9869b286","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:23:30.734 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.731+0000 7f2d1cac9640 10 librados: watch_flush enter 2024-03-25T18:23:30.734 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.731+0000 7f2d1cac9640 10 librados: watch_flush exit 2024-03-25T18:23:30.734 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.731+0000 7f2d1cac9640 20 client.4552.objecter shutdown clearing up homeless session... 2024-03-25T18:23:30.734 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.731+0000 7f2d1cac9640 10 client.4552.objecter successfully canceled tick 2024-03-25T18:23:30.734 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.731+0000 7f2d1cac9640 1 -- 172.21.15.97:0/3840413004 >> v1:172.21.15.98:6800/800387573 conn(0x7f2cf806e110 legacy=0x7f2cf80705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:30.734 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.731+0000 7f2d1cac9640 10 monclient: shutdown 2024-03-25T18:23:30.734 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.731+0000 7f2d1cac9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:23:30.734 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.731+0000 7f2d1cac9640 1 -- 172.21.15.97:0/3840413004 >> v1:172.21.15.98:6789/0 conn(0x7f2d180a98f0 legacy=0x7f2d18139370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:30.734 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.731+0000 7f2d1cac9640 1 -- 172.21.15.97:0/3840413004 shutdown_connections 2024-03-25T18:23:30.734 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.731+0000 7f2d1cac9640 1 -- 172.21.15.97:0/3840413004 >> 172.21.15.97:0/3840413004 conn(0x7f2d180b7ea0 msgr2=0x7f2d18074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:23:30.734 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.732+0000 7f2d1cac9640 1 -- 172.21.15.97:0/3840413004 shutdown_connections 2024-03-25T18:23:30.734 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.732+0000 7f2d1cac9640 1 -- 172.21.15.97:0/3840413004 wait complete. 2024-03-25T18:23:30.735 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:30.732+0000 7f2d1cac9640 1 librados: shutdown 2024-03-25T18:23:30.745 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:23:33.182 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:07:35.220 34931== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:33.235 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:07:35.273 34931== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:33.282 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:07:35.320 34931== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:33.289 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:07:35.328 34935== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:33.296 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:07:35.339 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:33.299 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:07:35.337 34935== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:33.302 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:07:35.345 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:33.327 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:07:35.370 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:33.333 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:07:35.372 34935== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:37.552 DEBUG:teuthology.orchestra.run.smithi097:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:23:37.664 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:37.662+0000 7f0feed32640 10 monclient: get_monmap_and_config 2024-03-25T18:23:37.664 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:37.662+0000 7f0feed32640 10 monclient: build_initial_monmap 2024-03-25T18:23:37.664 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:37.662+0000 7f0feed32640 10 monclient: monmap: 2024-03-25T18:23:37.664 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:23:37.664 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:23:37.664 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:23:37.664025+0000 2024-03-25T18:23:37.664 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:23:37.664025+0000 2024-03-25T18:23:37.664 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:23:37.664 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:23:37.664 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:23:37.664 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T18:23:37.665 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-b 2024-03-25T18:23:37.665 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:23:37.666 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:37.663+0000 7f0feed32640 1 Processor -- start 2024-03-25T18:23:37.666 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:37.663+0000 7f0feed32640 1 -- start start 2024-03-25T18:23:37.666 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:37.663+0000 7f0feed32640 10 monclient: init 2024-03-25T18:23:37.666 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:37.664+0000 7f0feed32640 10 monclient: _reopen_session rank -1 2024-03-25T18:23:37.666 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:37.664+0000 7f0feed32640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:23:37.666 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:37.664+0000 7f0feed32640 10 monclient(hunting): picked mon.noname-c con 0x7f0fe80a98f0 addr v1:172.21.15.98:6789/0 2024-03-25T18:23:37.666 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:37.664+0000 7f0feed32640 10 monclient(hunting): picked mon.noname-b con 0x7f0fe813f710 addr v1:172.21.15.97:6790/0 2024-03-25T18:23:37.666 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:37.664+0000 7f0feed32640 10 monclient(hunting): picked mon.noname-a con 0x7f0fe81430c0 addr v1:172.21.15.97:6789/0 2024-03-25T18:23:37.666 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:37.664+0000 7f0feed32640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fe8142520 con 0x7f0fe81430c0 2024-03-25T18:23:37.666 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:37.664+0000 7f0feed32640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fe80b28a0 con 0x7f0fe80a98f0 2024-03-25T18:23:37.666 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:37.664+0000 7f0feed32640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fe80b2a30 con 0x7f0fe813f710 2024-03-25T18:23:37.666 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:37.664+0000 7f0feed32640 10 monclient(hunting): _renew_subs 2024-03-25T18:23:37.666 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:37.664+0000 7f0feed32640 10 monclient(hunting): authenticate will time out at 8029.940065s 2024-03-25T18:23:37.677 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:37.676+0000 7f0fe7fff640 1 --1- >> v1:172.21.15.97:6790/0 conn(0x7f0fe813f710 0x7f0fe81429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6790/0 says I am v1:172.21.15.97:49148/0 (socket says 172.21.15.97:49148) 2024-03-25T18:23:37.677 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:37.676+0000 7f0fe7fff640 1 -- 172.21.15.97:0/3796072080 learned_addr learned my addr 172.21.15.97:0/3796072080 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:23:37.686 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:37.684+0000 7f0fe77fe640 1 -- 172.21.15.97:0/3796072080 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 417404919 0 0) 0x7f0fe80b2a30 con 0x7f0fe813f710 2024-03-25T18:23:37.686 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:37.684+0000 7f0fe77fe640 10 monclient(hunting): _negotiate 2024-03-25T18:23:37.686 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:37.684+0000 7f0fe77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:37.686 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:37.684+0000 7f0fe77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:37.686 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:37.684+0000 7f0fe77fe640 10 monclient(hunting): my global_id is 4544 2024-03-25T18:23:37.686 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:37.684+0000 7f0fe77fe640 1 -- 172.21.15.97:0/3796072080 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0fd8003520 con 0x7f0fe813f710 2024-03-25T18:23:37.693 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:37.691+0000 7f0fe77fe640 1 -- 172.21.15.97:0/3796072080 <== mon.2 v1:172.21.15.97:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1391575038 0 0) 0x7f0fd8003520 con 0x7f0fe813f710 2024-03-25T18:23:37.693 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:37.691+0000 7f0fe77fe640 1 -- 172.21.15.97:0/3796072080 >> v1:172.21.15.98:6789/0 conn(0x7f0fe80a98f0 legacy=0x7f0fe80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:37.693 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:37.691+0000 7f0fe77fe640 1 -- 172.21.15.97:0/3796072080 >> v1:172.21.15.97:6789/0 conn(0x7f0fe81430c0 legacy=0x7f0fe814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:37.693 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:37.691+0000 7f0fe77fe640 10 monclient: _finish_hunting 0 2024-03-25T18:23:37.693 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:37.691+0000 7f0fe77fe640 1 monclient: found mon.noname-b 2024-03-25T18:23:37.693 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:37.691+0000 7f0fe77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:23:37.693 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:37.691+0000 7f0fe77fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.97:6790/0 2024-03-25T18:23:37.693 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:37.691+0000 7f0fe77fe640 1 -- 172.21.15.97:0/3796072080 --> v1:172.21.15.97:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fe80b44c0 con 0x7f0fe813f710 2024-03-25T18:23:37.693 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:37.691+0000 7f0fe77fe640 10 monclient: _finish_auth 0 2024-03-25T18:23:37.693 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:37.691+0000 7f0fe77fe640 10 monclient: _check_auth_tickets 2024-03-25T18:23:37.694 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:37.691+0000 7f0fe77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:23:37.694 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:37.691+0000 7f0fe77fe640 1 -- 172.21.15.97:0/3796072080 <== mon.2 v1:172.21.15.97:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f0fd4001690 con 0x7f0fe813f710 2024-03-25T18:23:37.694 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:37.691+0000 7f0fe77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:37.694 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:37.691+0000 7f0fe77fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:23:37.694 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:37.691+0000 7f0fe77fe640 10 monclient: dump: 2024-03-25T18:23:37.694 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:23:37.694 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:23:37.694 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:23:37.694 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:23:37.694 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:37.694 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:23:37.694 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:23:37.694 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:23:37.694 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:23:37.695 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:23:37.695 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:37.691+0000 7f0feed32640 5 monclient: authenticate success, global_id 4544 2024-03-25T18:23:37.695 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:37.691+0000 7f0feed32640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:23:37.705 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:37.703+0000 7f0fe77fe640 1 -- 172.21.15.97:0/3796072080 <== mon.2 v1:172.21.15.97:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0fd4001870 con 0x7f0fe813f710 2024-03-25T18:23:37.705 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:37.703+0000 7f0fe77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:23:37.705 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:37.703+0000 7f0fe77fe640 1 -- 172.21.15.97:0/3796072080 <== mon.2 v1:172.21.15.97:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f0fd4001c80 con 0x7f0fe813f710 2024-03-25T18:23:37.705 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:37.703+0000 7f0feed32640 10 monclient: get_monmap_and_config success 2024-03-25T18:23:37.705 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:37.703+0000 7f0feed32640 4 set_mon_vals no callback set 2024-03-25T18:23:37.705 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:37.703+0000 7f0fe77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:37.705 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:37.703+0000 7f0fe77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:23:37.705 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:37.703+0000 7f0fe77fe640 10 monclient: dump: 2024-03-25T18:23:37.705 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:23:37.705 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:23:37.705 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:23:37.706 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:23:37.706 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:37.706 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:23:37.706 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:23:37.706 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:23:37.706 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:23:37.706 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:23:37.708 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:37.704+0000 7f0feed32640 10 monclient: shutdown 2024-03-25T18:23:37.708 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:37.704+0000 7f0feed32640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:23:37.708 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:37.704+0000 7f0feed32640 1 -- 172.21.15.97:0/3796072080 >> v1:172.21.15.97:6790/0 conn(0x7f0fe813f710 legacy=0x7f0fe81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:37.708 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:37.704+0000 7f0feed32640 1 -- 172.21.15.97:0/3796072080 shutdown_connections 2024-03-25T18:23:37.708 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:37.704+0000 7f0feed32640 1 -- 172.21.15.97:0/3796072080 >> 172.21.15.97:0/3796072080 conn(0x7f0fe80b7ea0 msgr2=0x7f0fe80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:23:37.708 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:37.704+0000 7f0feed32640 1 -- 172.21.15.97:0/3796072080 shutdown_connections 2024-03-25T18:23:37.708 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:37.704+0000 7f0feed32640 1 -- 172.21.15.97:0/3796072080 wait complete. 2024-03-25T18:23:37.708 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:37.705+0000 7f0feed32640 10 monclient: build_initial_monmap 2024-03-25T18:23:37.708 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:37.705+0000 7f0feed32640 10 monclient: monmap: 2024-03-25T18:23:37.709 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:23:37.709 INFO:teuthology.misc.health.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:23:37.709 INFO:teuthology.misc.health.smithi097.stderr:last_changed 0.000000 2024-03-25T18:23:37.709 INFO:teuthology.misc.health.smithi097.stderr:created 0.000000 2024-03-25T18:23:37.709 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:23:37.709 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 1 2024-03-25T18:23:37.709 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:23:37.709 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T18:23:37.709 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-c 2024-03-25T18:23:37.709 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:23:37.709 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:37.705+0000 7f0feed32640 1 librados: starting msgr at 2024-03-25T18:23:37.709 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:37.705+0000 7f0feed32640 1 librados: starting objecter 2024-03-25T18:23:37.709 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:37.705+0000 7f0feed32640 1 Processor -- start 2024-03-25T18:23:37.709 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:37.705+0000 7f0feed32640 1 -- start start 2024-03-25T18:23:37.709 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:37.705+0000 7f0feed32640 1 librados: setting wanted keys 2024-03-25T18:23:37.710 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:37.705+0000 7f0feed32640 1 librados: calling monclient init 2024-03-25T18:23:37.710 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:37.705+0000 7f0feed32640 10 monclient: init 2024-03-25T18:23:37.710 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:37.706+0000 7f0feed32640 10 monclient: _reopen_session rank -1 2024-03-25T18:23:37.710 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:37.706+0000 7f0feed32640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:23:37.710 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:37.706+0000 7f0feed32640 10 monclient(hunting): picked mon.noname-c con 0x7f0fe80a98f0 addr v1:172.21.15.97:6790/0 2024-03-25T18:23:37.710 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:37.706+0000 7f0feed32640 10 monclient(hunting): picked mon.noname-b con 0x7f0fe813f710 addr v1:172.21.15.98:6789/0 2024-03-25T18:23:37.710 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:37.706+0000 7f0feed32640 10 monclient(hunting): picked mon.noname-a con 0x7f0fe81430c0 addr v1:172.21.15.97:6789/0 2024-03-25T18:23:37.710 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:37.706+0000 7f0feed32640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fe8136f50 con 0x7f0fe81430c0 2024-03-25T18:23:37.710 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:37.706+0000 7f0feed32640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fe8138f30 con 0x7f0fe813f710 2024-03-25T18:23:37.710 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:37.706+0000 7f0feed32640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fe813c4d0 con 0x7f0fe80a98f0 2024-03-25T18:23:37.710 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:37.706+0000 7f0feed32640 10 monclient(hunting): _renew_subs 2024-03-25T18:23:37.710 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:37.706+0000 7f0feed32640 10 monclient(hunting): authenticate will time out at 8029.982490s 2024-03-25T18:23:37.716 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:37.714+0000 7f0fe7fff640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f0fe813f710 0x7f0fe8132f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.97:37420/0 (socket says 172.21.15.97:37420) 2024-03-25T18:23:37.716 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:37.714+0000 7f0fe7fff640 1 -- 172.21.15.97:0/3757860419 learned_addr learned my addr 172.21.15.97:0/3757860419 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:23:37.717 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:37.716+0000 7f0fe57fa640 10 client.?.objecter ms_handle_connect 0x7f0fe80a98f0 2024-03-25T18:23:37.717 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:37.716+0000 7f0fe57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:37.718 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:37.717+0000 7f0fe57fa640 10 client.?.objecter ms_handle_connect 0x7f0fe813f710 2024-03-25T18:23:37.718 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:37.717+0000 7f0fe57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:37.720 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:37.719+0000 7f0fe57fa640 10 client.?.objecter ms_handle_connect 0x7f0fe81430c0 2024-03-25T18:23:37.720 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:37.719+0000 7f0fe57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:37.728 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:37.727+0000 7f0fe57fa640 1 -- 172.21.15.97:0/3757860419 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1401157153 0 0) 0x7f0fe8138f30 con 0x7f0fe813f710 2024-03-25T18:23:37.728 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:37.727+0000 7f0fe57fa640 10 monclient(hunting): _negotiate 2024-03-25T18:23:37.729 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:37.727+0000 7f0fe57fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:37.729 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:37.727+0000 7f0fe57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:37.729 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:37.727+0000 7f0fe57fa640 10 monclient(hunting): my global_id is 4558 2024-03-25T18:23:37.729 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:37.727+0000 7f0fe57fa640 1 -- 172.21.15.97:0/3757860419 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0fcc004eb0 con 0x7f0fe813f710 2024-03-25T18:23:37.732 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:37.730+0000 7f0fe57fa640 1 -- 172.21.15.97:0/3757860419 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1038037189 0 0) 0x7f0fe813c4d0 con 0x7f0fe80a98f0 2024-03-25T18:23:37.732 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:37.730+0000 7f0fe57fa640 10 monclient(hunting): _negotiate 2024-03-25T18:23:37.732 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:37.730+0000 7f0fe57fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:37.732 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:37.730+0000 7f0fe57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:37.732 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:37.730+0000 7f0fe57fa640 10 monclient(hunting): my global_id is 4547 2024-03-25T18:23:37.732 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:37.730+0000 7f0fe57fa640 1 -- 172.21.15.97:0/3757860419 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0fe8138f30 con 0x7f0fe80a98f0 2024-03-25T18:23:37.735 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:37.733+0000 7f0fe57fa640 1 -- 172.21.15.97:0/3757860419 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3814518851 0 0) 0x7f0fe8136f50 con 0x7f0fe81430c0 2024-03-25T18:23:37.735 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:37.733+0000 7f0fe57fa640 10 monclient(hunting): _negotiate 2024-03-25T18:23:37.735 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:37.733+0000 7f0fe57fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:37.735 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:37.733+0000 7f0fe57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:37.735 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:37.733+0000 7f0fe57fa640 10 monclient(hunting): my global_id is 4551 2024-03-25T18:23:37.735 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:37.733+0000 7f0fe57fa640 1 -- 172.21.15.97:0/3757860419 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0fe813c4d0 con 0x7f0fe81430c0 2024-03-25T18:23:37.741 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:37.739+0000 7f0fe57fa640 1 -- 172.21.15.97:0/3757860419 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 283713547 0 0) 0x7f0fcc004eb0 con 0x7f0fe813f710 2024-03-25T18:23:37.741 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:37.739+0000 7f0fe57fa640 1 -- 172.21.15.97:0/3757860419 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0fe8136f50 con 0x7f0fe813f710 2024-03-25T18:23:37.741 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:37.739+0000 7f0fe57fa640 1 -- 172.21.15.97:0/3757860419 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f0fd4001bc0 con 0x7f0fe813f710 2024-03-25T18:23:37.741 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:37.739+0000 7f0fe57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:37.741 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:37.739+0000 7f0fe57fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:23:37.741 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:37.739+0000 7f0fe57fa640 10 monclient(hunting): dump: 2024-03-25T18:23:37.741 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:23:37.741 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:23:37.741 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:23:37.741 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:23:37.741 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:37.742 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:23:37.742 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:23:37.742 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:23:37.742 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:23:37.742 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:23:37.742 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:37.739+0000 7f0fe57fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:23:37.750 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:37.748+0000 7f0fe57fa640 1 -- 172.21.15.97:0/3757860419 <== mon.1 v1:172.21.15.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 76197480 0 0) 0x7f0fe8136f50 con 0x7f0fe813f710 2024-03-25T18:23:37.750 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:37.748+0000 7f0fe57fa640 1 -- 172.21.15.97:0/3757860419 >> v1:172.21.15.97:6790/0 conn(0x7f0fe80a98f0 legacy=0x7f0fe81da8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:37.750 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:37.748+0000 7f0fe57fa640 1 -- 172.21.15.97:0/3757860419 >> v1:172.21.15.97:6789/0 conn(0x7f0fe81430c0 legacy=0x7f0fe81e2c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:37.750 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:37.748+0000 7f0fe57fa640 10 monclient: _finish_hunting 0 2024-03-25T18:23:37.750 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:37.748+0000 7f0fe57fa640 1 monclient: found mon.b 2024-03-25T18:23:37.750 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:37.748+0000 7f0fe57fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:23:37.751 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:37.748+0000 7f0fe57fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:23:37.751 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:37.748+0000 7f0fe57fa640 1 -- 172.21.15.97:0/3757860419 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fe813c660 con 0x7f0fe813f710 2024-03-25T18:23:37.751 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:37.748+0000 7f0fe57fa640 10 monclient: _finish_auth 0 2024-03-25T18:23:37.751 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:37.748+0000 7f0fe57fa640 10 monclient: _check_auth_tickets 2024-03-25T18:23:37.751 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:37.748+0000 7f0fe57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:23:37.751 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:37.748+0000 7f0feed32640 5 monclient: authenticate success, global_id 4558 2024-03-25T18:23:37.751 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:37.748+0000 7f0feed32640 10 monclient: _renew_subs 2024-03-25T18:23:37.751 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:37.748+0000 7f0feed32640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:23:37.751 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:37.748+0000 7f0feed32640 1 -- 172.21.15.97:0/3757860419 --> v1:172.21.15.98:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fe8139650 con 0x7f0fe813f710 2024-03-25T18:23:37.751 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:37.748+0000 7f0feed32640 10 client.4558.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:23:37.751 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:37.748+0000 7f0feed32640 10 monclient: _renew_subs 2024-03-25T18:23:37.751 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:37.748+0000 7f0feed32640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:23:37.751 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:37.748+0000 7f0feed32640 1 -- 172.21.15.97:0/3757860419 --> v1:172.21.15.98:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fe81336a0 con 0x7f0fe813f710 2024-03-25T18:23:37.751 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:37.748+0000 7f0feed32640 1 librados: init done 2024-03-25T18:23:37.752 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:37.749+0000 7f0feed32640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:37.752 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:37.749+0000 7f0feed32640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:37.752 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:37.749+0000 7f0feed32640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:23:37.752 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:37.749+0000 7f0feed32640 1 -- 172.21.15.97:0/3757860419 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fe80627b0 con 0x7f0fe813f710 2024-03-25T18:23:38.028 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:38.026+0000 7f0fe57fa640 1 -- 172.21.15.97:0/3757860419 <== mon.1 v1:172.21.15.98:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0fd4004bf0 con 0x7f0fe813f710 2024-03-25T18:23:38.028 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:38.026+0000 7f0fe57fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:23:38.028 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:38.026+0000 7f0fe57fa640 1 -- 172.21.15.97:0/3757860419 <== mon.1 v1:172.21.15.98:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f0fd4001250 con 0x7f0fe813f710 2024-03-25T18:23:38.028 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:38.026+0000 7f0fe57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:38.028 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:38.026+0000 7f0fe57fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:23:38.028 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:38.026+0000 7f0fe57fa640 10 monclient: dump: 2024-03-25T18:23:38.028 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:23:38.028 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:23:38.028 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:23:38.028 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:23:38.029 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:38.029 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:23:38.029 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:23:38.029 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:23:38.029 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:23:38.029 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:23:38.029 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:38.026+0000 7f0fe6ffd640 4 set_mon_vals no callback set 2024-03-25T18:23:38.029 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:38.027+0000 7f0fe57fa640 1 -- 172.21.15.97:0/3757860419 <== mon.1 v1:172.21.15.98:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2377622137 0 0) 0x7f0fd401ad70 con 0x7f0fe813f710 2024-03-25T18:23:38.030 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:38.028+0000 7f0fecace640 10 monclient: get_auth_request con 0x7f0fcc06ddf0 auth_method 0 2024-03-25T18:23:38.030 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:38.028+0000 7f0fe57fa640 10 client.4558.objecter ms_handle_connect 0x7f0fcc06ddf0 2024-03-25T18:23:38.034 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:38.032+0000 7f0fe57fa640 1 -- 172.21.15.97:0/3757860419 <== mon.1 v1:172.21.15.98:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 430711918 0 0) 0x7f0fd4054a00 con 0x7f0fe813f710 2024-03-25T18:23:38.034 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:38.032+0000 7f0fe57fa640 10 client.4558.objecter ms_dispatch 0x7f0fe8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:23:38.034 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:38.032+0000 7f0fe57fa640 3 client.4558.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:23:38.034 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:38.032+0000 7f0fe57fa640 3 client.4558.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:23:38.034 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:38.032+0000 7f0fe57fa640 20 client.4558.objecter dump_active .. 0 homeless 2024-03-25T18:23:38.034 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:38.032+0000 7f0fe57fa640 1 -- 172.21.15.97:0/3757860419 <== mon.1 v1:172.21.15.98:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0fd4050340 con 0x7f0fe813f710 2024-03-25T18:23:38.034 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:38.032+0000 7f0fe57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:38.034 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:38.032+0000 7f0fe57fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:23:38.284 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:38.282+0000 7f0feed32640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:38.284 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:38.282+0000 7f0feed32640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:38.284 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:38.282+0000 7f0feed32640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:23:38.284 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:38.282+0000 7f0feed32640 1 -- 172.21.15.97:0/3757860419 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0fe8069f40 con 0x7f0fe813f710 2024-03-25T18:23:38.307 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:38.305+0000 7f0fe57fa640 1 -- 172.21.15.97:0/3757860419 <== mon.1 v1:172.21.15.98:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2169427412) 0x7f0fd4052440 con 0x7f0fe813f710 2024-03-25T18:23:38.307 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:38.305+0000 7f0fe57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:38.307 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:38.305+0000 7f0fe57fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:23:38.307 INFO:teuthology.misc.health.smithi097.stdout: 2024-03-25T18:23:38.307 INFO:teuthology.misc.health.smithi097.stdout:{"epoch":10,"fsid":"4c3d472c-79b8-486b-9e3d-2738d4306754","created":"2024-03-25T18:16:14.737309+0000","modified":"2024-03-25T18:16:20.357492+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:20.357492+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cb5152b9-262f-45c3-94b8-18d0e81df558","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6daf8793-0838-473d-a7d6-9a0d2894c75d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"136f1493-3b9c-4f43-a2e7-641482d69dae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"397d538c-4f64-4daf-9e8b-662d09f91327","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4f1df08-663f-4bda-93f8-3878ab7c24b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c4b18d70-a3ce-4b15-9b70-eae5525bd761","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8dd4033a-de0e-4f8a-96dc-59f3c1f825b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0077722e-ffa6-4a3d-9891-929d9869b286","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:23:38.310 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:38.308+0000 7f0feed32640 10 librados: watch_flush enter 2024-03-25T18:23:38.310 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:38.308+0000 7f0feed32640 10 librados: watch_flush exit 2024-03-25T18:23:38.310 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:38.308+0000 7f0feed32640 20 client.4558.objecter shutdown clearing up homeless session... 2024-03-25T18:23:38.310 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:38.308+0000 7f0feed32640 10 client.4558.objecter successfully canceled tick 2024-03-25T18:23:38.310 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:38.308+0000 7f0feed32640 1 -- 172.21.15.97:0/3757860419 >> v1:172.21.15.98:6800/800387573 conn(0x7f0fcc06ddf0 legacy=0x7f0fcc0702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:38.310 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:38.308+0000 7f0feed32640 10 monclient: shutdown 2024-03-25T18:23:38.310 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:38.308+0000 7f0feed32640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:23:38.310 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:38.308+0000 7f0feed32640 1 -- 172.21.15.97:0/3757860419 >> v1:172.21.15.98:6789/0 conn(0x7f0fe813f710 legacy=0x7f0fe8132f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:38.311 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:38.308+0000 7f0feed32640 1 -- 172.21.15.97:0/3757860419 shutdown_connections 2024-03-25T18:23:38.311 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:38.308+0000 7f0feed32640 1 -- 172.21.15.97:0/3757860419 >> 172.21.15.97:0/3757860419 conn(0x7f0fe80b7ea0 msgr2=0x7f0fe8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:23:38.311 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:38.308+0000 7f0feed32640 1 -- 172.21.15.97:0/3757860419 shutdown_connections 2024-03-25T18:23:38.311 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:38.308+0000 7f0feed32640 1 -- 172.21.15.97:0/3757860419 wait complete. 2024-03-25T18:23:38.311 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:38.308+0000 7f0feed32640 1 librados: shutdown 2024-03-25T18:23:38.321 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:23:45.128 DEBUG:teuthology.orchestra.run.smithi097:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:23:45.244 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.242+0000 7efd40b8f640 10 monclient: get_monmap_and_config 2024-03-25T18:23:45.244 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.242+0000 7efd40b8f640 10 monclient: build_initial_monmap 2024-03-25T18:23:45.244 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.242+0000 7efd40b8f640 10 monclient: monmap: 2024-03-25T18:23:45.244 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:23:45.244 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:23:45.244 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:23:45.243989+0000 2024-03-25T18:23:45.244 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:23:45.243989+0000 2024-03-25T18:23:45.244 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:23:45.244 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:23:45.245 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:23:45.245 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T18:23:45.245 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-b 2024-03-25T18:23:45.245 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:23:45.246 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.243+0000 7efd40b8f640 1 Processor -- start 2024-03-25T18:23:45.246 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.243+0000 7efd40b8f640 1 -- start start 2024-03-25T18:23:45.246 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.243+0000 7efd40b8f640 10 monclient: init 2024-03-25T18:23:45.246 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.244+0000 7efd40b8f640 10 monclient: _reopen_session rank -1 2024-03-25T18:23:45.246 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.244+0000 7efd40b8f640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:23:45.247 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.244+0000 7efd40b8f640 10 monclient(hunting): picked mon.noname-c con 0x7efd3c0a98f0 addr v1:172.21.15.98:6789/0 2024-03-25T18:23:45.247 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.244+0000 7efd40b8f640 10 monclient(hunting): picked mon.noname-a con 0x7efd3c13f710 addr v1:172.21.15.97:6789/0 2024-03-25T18:23:45.247 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.244+0000 7efd40b8f640 10 monclient(hunting): picked mon.noname-b con 0x7efd3c1430c0 addr v1:172.21.15.97:6790/0 2024-03-25T18:23:45.247 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.244+0000 7efd40b8f640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd3c142520 con 0x7efd3c13f710 2024-03-25T18:23:45.247 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.244+0000 7efd40b8f640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd3c0b28a0 con 0x7efd3c0a98f0 2024-03-25T18:23:45.247 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.244+0000 7efd40b8f640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd3c0b2a30 con 0x7efd3c1430c0 2024-03-25T18:23:45.247 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.244+0000 7efd40b8f640 10 monclient(hunting): _renew_subs 2024-03-25T18:23:45.247 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.244+0000 7efd40b8f640 10 monclient(hunting): authenticate will time out at 8037.520143s 2024-03-25T18:23:45.254 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.252+0000 7efd3a59c640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7efd3c0a98f0 0x7efd3c0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.97:41978/0 (socket says 172.21.15.97:41978) 2024-03-25T18:23:45.254 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.252+0000 7efd3a59c640 1 -- 172.21.15.97:0/1894865863 learned_addr learned my addr 172.21.15.97:0/1894865863 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:23:45.264 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.262+0000 7efd3959a640 1 -- 172.21.15.97:0/1894865863 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 328782049 0 0) 0x7efd3c0b28a0 con 0x7efd3c0a98f0 2024-03-25T18:23:45.264 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.262+0000 7efd3959a640 10 monclient(hunting): _negotiate 2024-03-25T18:23:45.264 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.262+0000 7efd3959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:45.264 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.262+0000 7efd3959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:45.264 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.262+0000 7efd3959a640 10 monclient(hunting): my global_id is 4561 2024-03-25T18:23:45.264 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.262+0000 7efd3959a640 1 -- 172.21.15.97:0/1894865863 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd24003520 con 0x7efd3c0a98f0 2024-03-25T18:23:45.278 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.276+0000 7efd3959a640 1 -- 172.21.15.97:0/1894865863 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4174627705 0 0) 0x7efd3c142520 con 0x7efd3c13f710 2024-03-25T18:23:45.278 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.276+0000 7efd3959a640 10 monclient(hunting): _negotiate 2024-03-25T18:23:45.278 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.276+0000 7efd3959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:45.278 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.276+0000 7efd3959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:45.278 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.276+0000 7efd3959a640 10 monclient(hunting): my global_id is 4554 2024-03-25T18:23:45.278 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.276+0000 7efd3959a640 1 -- 172.21.15.97:0/1894865863 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd3c0b28a0 con 0x7efd3c13f710 2024-03-25T18:23:45.280 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.278+0000 7efd3959a640 1 -- 172.21.15.97:0/1894865863 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1520808767 0 0) 0x7efd24003520 con 0x7efd3c0a98f0 2024-03-25T18:23:45.280 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.278+0000 7efd3959a640 1 -- 172.21.15.97:0/1894865863 >> v1:172.21.15.97:6790/0 conn(0x7efd3c1430c0 legacy=0x7efd3c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:45.280 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.278+0000 7efd3959a640 1 -- 172.21.15.97:0/1894865863 >> v1:172.21.15.97:6789/0 conn(0x7efd3c13f710 legacy=0x7efd3c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:45.280 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.278+0000 7efd3959a640 10 monclient: _finish_hunting 0 2024-03-25T18:23:45.280 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.278+0000 7efd3959a640 1 monclient: found mon.noname-c 2024-03-25T18:23:45.280 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.278+0000 7efd3959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:23:45.281 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.278+0000 7efd3959a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.98:6789/0 2024-03-25T18:23:45.281 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.278+0000 7efd3959a640 1 -- 172.21.15.97:0/1894865863 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd3c0b44c0 con 0x7efd3c0a98f0 2024-03-25T18:23:45.281 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.278+0000 7efd3959a640 10 monclient: _finish_auth 0 2024-03-25T18:23:45.281 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.278+0000 7efd3959a640 10 monclient: _check_auth_tickets 2024-03-25T18:23:45.281 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.278+0000 7efd3959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:23:45.281 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.278+0000 7efd3959a640 1 -- 172.21.15.97:0/1894865863 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7efd34001690 con 0x7efd3c0a98f0 2024-03-25T18:23:45.281 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.278+0000 7efd3959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:45.281 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.278+0000 7efd3959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:23:45.281 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.278+0000 7efd3959a640 10 monclient: dump: 2024-03-25T18:23:45.281 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:23:45.281 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:23:45.281 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:23:45.281 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:23:45.281 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:45.281 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:23:45.282 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:23:45.282 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:23:45.282 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:23:45.282 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:23:45.282 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.278+0000 7efd40b8f640 5 monclient: authenticate success, global_id 4561 2024-03-25T18:23:45.282 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.278+0000 7efd40b8f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:23:45.297 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.296+0000 7efd3959a640 1 -- 172.21.15.97:0/1894865863 <== mon.1 v1:172.21.15.98:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efd340017f0 con 0x7efd3c0a98f0 2024-03-25T18:23:45.297 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.296+0000 7efd3959a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:23:45.297 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.296+0000 7efd3959a640 1 -- 172.21.15.97:0/1894865863 <== mon.1 v1:172.21.15.98:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7efd34001c00 con 0x7efd3c0a98f0 2024-03-25T18:23:45.297 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.296+0000 7efd40b8f640 10 monclient: get_monmap_and_config success 2024-03-25T18:23:45.297 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.296+0000 7efd40b8f640 4 set_mon_vals no callback set 2024-03-25T18:23:45.297 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.296+0000 7efd3959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:45.297 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.296+0000 7efd3959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:23:45.298 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.296+0000 7efd3959a640 10 monclient: dump: 2024-03-25T18:23:45.298 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:23:45.298 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:23:45.298 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:23:45.298 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:23:45.298 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:45.298 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:23:45.298 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:23:45.298 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:23:45.298 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:23:45.298 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:23:45.299 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.296+0000 7efd40b8f640 10 monclient: shutdown 2024-03-25T18:23:45.300 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.296+0000 7efd40b8f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:23:45.300 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.296+0000 7efd40b8f640 1 -- 172.21.15.97:0/1894865863 >> v1:172.21.15.98:6789/0 conn(0x7efd3c0a98f0 legacy=0x7efd3c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:45.300 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.297+0000 7efd40b8f640 1 -- 172.21.15.97:0/1894865863 shutdown_connections 2024-03-25T18:23:45.300 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.297+0000 7efd40b8f640 1 -- 172.21.15.97:0/1894865863 >> 172.21.15.97:0/1894865863 conn(0x7efd3c0b7ea0 msgr2=0x7efd3c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:23:45.300 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.297+0000 7efd40b8f640 1 -- 172.21.15.97:0/1894865863 shutdown_connections 2024-03-25T18:23:45.300 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.297+0000 7efd40b8f640 1 -- 172.21.15.97:0/1894865863 wait complete. 2024-03-25T18:23:45.300 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.298+0000 7efd40b8f640 10 monclient: build_initial_monmap 2024-03-25T18:23:45.300 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.298+0000 7efd40b8f640 10 monclient: monmap: 2024-03-25T18:23:45.300 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:23:45.300 INFO:teuthology.misc.health.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:23:45.300 INFO:teuthology.misc.health.smithi097.stderr:last_changed 0.000000 2024-03-25T18:23:45.300 INFO:teuthology.misc.health.smithi097.stderr:created 0.000000 2024-03-25T18:23:45.300 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:23:45.300 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 1 2024-03-25T18:23:45.301 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:23:45.301 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T18:23:45.301 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-c 2024-03-25T18:23:45.301 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:23:45.301 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.298+0000 7efd40b8f640 1 librados: starting msgr at 2024-03-25T18:23:45.301 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.298+0000 7efd40b8f640 1 librados: starting objecter 2024-03-25T18:23:45.302 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.298+0000 7efd40b8f640 1 Processor -- start 2024-03-25T18:23:45.302 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.299+0000 7efd40b8f640 1 -- start start 2024-03-25T18:23:45.302 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.299+0000 7efd40b8f640 1 librados: setting wanted keys 2024-03-25T18:23:45.302 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.299+0000 7efd40b8f640 1 librados: calling monclient init 2024-03-25T18:23:45.302 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.299+0000 7efd40b8f640 10 monclient: init 2024-03-25T18:23:45.302 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.299+0000 7efd40b8f640 10 monclient: _reopen_session rank -1 2024-03-25T18:23:45.302 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.299+0000 7efd40b8f640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:23:45.302 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.299+0000 7efd40b8f640 10 monclient(hunting): picked mon.noname-a con 0x7efd3c0a98f0 addr v1:172.21.15.97:6789/0 2024-03-25T18:23:45.303 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.299+0000 7efd40b8f640 10 monclient(hunting): picked mon.noname-b con 0x7efd3c13f710 addr v1:172.21.15.98:6789/0 2024-03-25T18:23:45.303 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.299+0000 7efd40b8f640 10 monclient(hunting): picked mon.noname-c con 0x7efd3c1430c0 addr v1:172.21.15.97:6790/0 2024-03-25T18:23:45.303 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.299+0000 7efd40b8f640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd3c139990 con 0x7efd3c0a98f0 2024-03-25T18:23:45.303 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.299+0000 7efd40b8f640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd3c0abb50 con 0x7efd3c13f710 2024-03-25T18:23:45.303 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.299+0000 7efd40b8f640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd3c0abe30 con 0x7efd3c1430c0 2024-03-25T18:23:45.303 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.299+0000 7efd40b8f640 10 monclient(hunting): _renew_subs 2024-03-25T18:23:45.303 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.299+0000 7efd40b8f640 10 monclient(hunting): authenticate will time out at 8037.575781s 2024-03-25T18:23:45.310 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.309+0000 7efd3ad9d640 1 --1- >> v1:172.21.15.97:6790/0 conn(0x7efd3c1430c0 0x7efd3c138300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.97:6790/0 says I am v1:172.21.15.97:38258/0 (socket says 172.21.15.97:38258) 2024-03-25T18:23:45.310 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.309+0000 7efd3ad9d640 1 -- 172.21.15.97:0/2937253522 learned_addr learned my addr 172.21.15.97:0/2937253522 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:23:45.311 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.309+0000 7efd2affd640 10 client.?.objecter ms_handle_connect 0x7efd3c1430c0 2024-03-25T18:23:45.311 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.309+0000 7efd2affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:45.316 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.315+0000 7efd2affd640 10 client.?.objecter ms_handle_connect 0x7efd3c0a98f0 2024-03-25T18:23:45.316 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.315+0000 7efd2affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:45.323 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.322+0000 7efd2affd640 10 client.?.objecter ms_handle_connect 0x7efd3c13f710 2024-03-25T18:23:45.323 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.322+0000 7efd2affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:45.324 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.323+0000 7efd2affd640 1 -- 172.21.15.97:0/2937253522 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3737730111 0 0) 0x7efd3c0abe30 con 0x7efd3c1430c0 2024-03-25T18:23:45.324 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.323+0000 7efd2affd640 10 monclient(hunting): _negotiate 2024-03-25T18:23:45.324 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.323+0000 7efd2affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:45.324 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.323+0000 7efd2affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:45.324 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.323+0000 7efd2affd640 10 monclient(hunting): my global_id is 4553 2024-03-25T18:23:45.325 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.323+0000 7efd2affd640 1 -- 172.21.15.97:0/2937253522 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd1c004eb0 con 0x7efd3c1430c0 2024-03-25T18:23:45.340 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.339+0000 7efd2affd640 1 -- 172.21.15.97:0/2937253522 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 868045455 0 0) 0x7efd3c0abb50 con 0x7efd3c13f710 2024-03-25T18:23:45.340 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.339+0000 7efd2affd640 10 monclient(hunting): _negotiate 2024-03-25T18:23:45.340 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.339+0000 7efd2affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:45.340 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.339+0000 7efd2affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:45.340 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.339+0000 7efd2affd640 10 monclient(hunting): my global_id is 4564 2024-03-25T18:23:45.341 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.339+0000 7efd2affd640 1 -- 172.21.15.97:0/2937253522 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd3c0abe30 con 0x7efd3c13f710 2024-03-25T18:23:45.343 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.341+0000 7efd2affd640 1 -- 172.21.15.97:0/2937253522 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2647625911 0 0) 0x7efd3c139990 con 0x7efd3c0a98f0 2024-03-25T18:23:45.343 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.341+0000 7efd2affd640 10 monclient(hunting): _negotiate 2024-03-25T18:23:45.343 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.341+0000 7efd2affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:45.343 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.341+0000 7efd2affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:45.343 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.341+0000 7efd2affd640 10 monclient(hunting): my global_id is 4557 2024-03-25T18:23:45.343 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.341+0000 7efd2affd640 1 -- 172.21.15.97:0/2937253522 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd3c0abb50 con 0x7efd3c0a98f0 2024-03-25T18:23:45.356 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.354+0000 7efd2affd640 1 -- 172.21.15.97:0/2937253522 <== mon.2 v1:172.21.15.97:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3121153054 0 0) 0x7efd1c004eb0 con 0x7efd3c1430c0 2024-03-25T18:23:45.356 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.354+0000 7efd2affd640 1 -- 172.21.15.97:0/2937253522 --> v1:172.21.15.97:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efd3c139990 con 0x7efd3c1430c0 2024-03-25T18:23:45.356 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.354+0000 7efd2affd640 1 -- 172.21.15.97:0/2937253522 <== mon.2 v1:172.21.15.97:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7efd2c001910 con 0x7efd3c1430c0 2024-03-25T18:23:45.356 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.354+0000 7efd2affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:45.356 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.354+0000 7efd2affd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:23:45.356 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.354+0000 7efd2affd640 10 monclient(hunting): dump: 2024-03-25T18:23:45.356 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:23:45.356 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:23:45.356 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:23:45.356 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:23:45.356 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:45.356 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:23:45.356 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:23:45.357 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:23:45.357 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:23:45.357 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:23:45.357 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.354+0000 7efd2affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:23:45.374 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.372+0000 7efd2affd640 1 -- 172.21.15.97:0/2937253522 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3966676975 0 0) 0x7efd3c0abb50 con 0x7efd3c0a98f0 2024-03-25T18:23:45.374 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.372+0000 7efd2affd640 1 -- 172.21.15.97:0/2937253522 --> v1:172.21.15.97:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efd1c004eb0 con 0x7efd3c0a98f0 2024-03-25T18:23:45.374 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.372+0000 7efd2affd640 1 -- 172.21.15.97:0/2937253522 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7efd34001b00 con 0x7efd3c0a98f0 2024-03-25T18:23:45.374 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.372+0000 7efd2affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:45.374 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.372+0000 7efd2affd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:23:45.374 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.372+0000 7efd2affd640 10 monclient(hunting): dump: 2024-03-25T18:23:45.374 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:23:45.375 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:23:45.375 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:23:45.375 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:23:45.375 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:45.375 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:23:45.375 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:23:45.375 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:23:45.375 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:23:45.375 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:23:45.378 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.376+0000 7efd2affd640 1 -- 172.21.15.97:0/2937253522 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2718619485 0 0) 0x7efd3c0abe30 con 0x7efd3c13f710 2024-03-25T18:23:45.378 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.376+0000 7efd2affd640 1 -- 172.21.15.97:0/2937253522 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efd3c0abb50 con 0x7efd3c13f710 2024-03-25T18:23:45.378 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.376+0000 7efd2affd640 1 -- 172.21.15.97:0/2937253522 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7efd30001cb0 con 0x7efd3c13f710 2024-03-25T18:23:45.378 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.376+0000 7efd2affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:45.378 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.376+0000 7efd2affd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:23:45.378 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.376+0000 7efd2affd640 10 monclient(hunting): dump: 2024-03-25T18:23:45.378 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:23:45.379 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:23:45.379 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:23:45.379 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:23:45.379 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:45.379 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:23:45.379 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:23:45.379 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:23:45.379 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:23:45.379 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:23:45.380 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.378+0000 7efd2affd640 1 -- 172.21.15.97:0/2937253522 <== mon.2 v1:172.21.15.97:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1917056924 0 0) 0x7efd3c139990 con 0x7efd3c1430c0 2024-03-25T18:23:45.381 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.378+0000 7efd2affd640 1 -- 172.21.15.97:0/2937253522 >> v1:172.21.15.98:6789/0 conn(0x7efd3c13f710 legacy=0x7efd3c136350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:45.381 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.378+0000 7efd2affd640 1 -- 172.21.15.97:0/2937253522 >> v1:172.21.15.97:6789/0 conn(0x7efd3c0a98f0 legacy=0x7efd3c0ac250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:45.381 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.378+0000 7efd2affd640 10 monclient: _finish_hunting 0 2024-03-25T18:23:45.381 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.378+0000 7efd2affd640 1 monclient: found mon.c 2024-03-25T18:23:45.381 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.378+0000 7efd2affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:23:45.381 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.378+0000 7efd2affd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.97:6790/0 2024-03-25T18:23:45.381 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.378+0000 7efd2affd640 1 -- 172.21.15.97:0/2937253522 --> v1:172.21.15.97:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd3c138dd0 con 0x7efd3c1430c0 2024-03-25T18:23:45.381 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.378+0000 7efd2affd640 10 monclient: _finish_auth 0 2024-03-25T18:23:45.381 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.378+0000 7efd2affd640 10 monclient: _check_auth_tickets 2024-03-25T18:23:45.381 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.378+0000 7efd2affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:23:45.381 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.378+0000 7efd40b8f640 5 monclient: authenticate success, global_id 4553 2024-03-25T18:23:45.381 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.378+0000 7efd40b8f640 10 monclient: _renew_subs 2024-03-25T18:23:45.381 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.378+0000 7efd40b8f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.97:6790/0 2024-03-25T18:23:45.381 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.378+0000 7efd40b8f640 1 -- 172.21.15.97:0/2937253522 --> v1:172.21.15.97:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd3c137ad0 con 0x7efd3c1430c0 2024-03-25T18:23:45.382 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.378+0000 7efd40b8f640 10 client.4553.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:23:45.382 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.378+0000 7efd40b8f640 10 monclient: _renew_subs 2024-03-25T18:23:45.382 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.378+0000 7efd40b8f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.97:6790/0 2024-03-25T18:23:45.382 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.378+0000 7efd40b8f640 1 -- 172.21.15.97:0/2937253522 --> v1:172.21.15.97:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efd3c1e32a0 con 0x7efd3c1430c0 2024-03-25T18:23:45.382 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.378+0000 7efd40b8f640 1 librados: init done 2024-03-25T18:23:45.382 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.379+0000 7efd40b8f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:45.382 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.379+0000 7efd40b8f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:45.382 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.379+0000 7efd40b8f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.97:6790/0 2024-03-25T18:23:45.382 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.379+0000 7efd40b8f640 1 -- 172.21.15.97:0/2937253522 --> v1:172.21.15.97:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd3c0627b0 con 0x7efd3c1430c0 2024-03-25T18:23:45.702 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.700+0000 7efd2affd640 1 -- 172.21.15.97:0/2937253522 <== mon.2 v1:172.21.15.97:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efd2c003610 con 0x7efd3c1430c0 2024-03-25T18:23:45.702 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.700+0000 7efd2affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:23:45.702 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.700+0000 7efd2affd640 1 -- 172.21.15.97:0/2937253522 <== mon.2 v1:172.21.15.97:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7efd2c0039e0 con 0x7efd3c1430c0 2024-03-25T18:23:45.702 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.700+0000 7efd2affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:45.702 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.700+0000 7efd2affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:23:45.702 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.700+0000 7efd2affd640 10 monclient: dump: 2024-03-25T18:23:45.702 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:23:45.702 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:23:45.702 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:23:45.702 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:23:45.702 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:45.703 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:23:45.703 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:23:45.703 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:23:45.703 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:23:45.703 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:23:45.703 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.700+0000 7efd38d99640 4 set_mon_vals no callback set 2024-03-25T18:23:45.703 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.701+0000 7efd2affd640 1 -- 172.21.15.97:0/2937253522 <== mon.2 v1:172.21.15.97:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2377622137 0 0) 0x7efd2c003c00 con 0x7efd3c1430c0 2024-03-25T18:23:45.703 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.702+0000 7efd3a59c640 10 monclient: get_auth_request con 0x7efd1c06e110 auth_method 0 2024-03-25T18:23:45.704 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.702+0000 7efd2affd640 10 client.4553.objecter ms_handle_connect 0x7efd1c06e110 2024-03-25T18:23:45.709 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.707+0000 7efd2affd640 1 -- 172.21.15.97:0/2937253522 <== mon.2 v1:172.21.15.97:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 430711918 0 0) 0x7efd2c054780 con 0x7efd3c1430c0 2024-03-25T18:23:45.709 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.707+0000 7efd2affd640 10 client.4553.objecter ms_dispatch 0x7efd3c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:23:45.709 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.707+0000 7efd2affd640 3 client.4553.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:23:45.709 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.707+0000 7efd2affd640 3 client.4553.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:23:45.709 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.707+0000 7efd2affd640 20 client.4553.objecter dump_active .. 0 homeless 2024-03-25T18:23:45.709 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.707+0000 7efd2affd640 1 -- 172.21.15.97:0/2937253522 <== mon.2 v1:172.21.15.97:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7efd2c050290 con 0x7efd3c1430c0 2024-03-25T18:23:45.709 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.707+0000 7efd2affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:45.709 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.707+0000 7efd2affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:23:45.939 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.937+0000 7efd40b8f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:45.939 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.937+0000 7efd40b8f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:45.939 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.937+0000 7efd40b8f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.97:6790/0 2024-03-25T18:23:45.939 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:45.937+0000 7efd40b8f640 1 -- 172.21.15.97:0/2937253522 --> v1:172.21.15.97:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efd3c138f60 con 0x7efd3c1430c0 2024-03-25T18:23:46.040 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:46.038+0000 7efd2affd640 1 -- 172.21.15.97:0/2937253522 <== mon.2 v1:172.21.15.97:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2169427412) 0x7efd2c0521c0 con 0x7efd3c1430c0 2024-03-25T18:23:46.040 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:46.038+0000 7efd2affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:46.040 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:46.038+0000 7efd2affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:23:46.040 INFO:teuthology.misc.health.smithi097.stdout: 2024-03-25T18:23:46.040 INFO:teuthology.misc.health.smithi097.stdout:{"epoch":10,"fsid":"4c3d472c-79b8-486b-9e3d-2738d4306754","created":"2024-03-25T18:16:14.737309+0000","modified":"2024-03-25T18:16:20.357492+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:20.357492+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cb5152b9-262f-45c3-94b8-18d0e81df558","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6daf8793-0838-473d-a7d6-9a0d2894c75d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"136f1493-3b9c-4f43-a2e7-641482d69dae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"397d538c-4f64-4daf-9e8b-662d09f91327","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4f1df08-663f-4bda-93f8-3878ab7c24b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c4b18d70-a3ce-4b15-9b70-eae5525bd761","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8dd4033a-de0e-4f8a-96dc-59f3c1f825b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0077722e-ffa6-4a3d-9891-929d9869b286","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:23:46.043 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:46.041+0000 7efd40b8f640 10 librados: watch_flush enter 2024-03-25T18:23:46.043 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:46.041+0000 7efd40b8f640 10 librados: watch_flush exit 2024-03-25T18:23:46.043 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:46.041+0000 7efd40b8f640 20 client.4553.objecter shutdown clearing up homeless session... 2024-03-25T18:23:46.043 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:46.041+0000 7efd40b8f640 10 client.4553.objecter successfully canceled tick 2024-03-25T18:23:46.043 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:46.041+0000 7efd40b8f640 1 -- 172.21.15.97:0/2937253522 >> v1:172.21.15.98:6800/800387573 conn(0x7efd1c06e110 legacy=0x7efd1c0705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:46.043 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:46.041+0000 7efd40b8f640 10 monclient: shutdown 2024-03-25T18:23:46.043 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:46.041+0000 7efd40b8f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:23:46.043 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:46.041+0000 7efd40b8f640 1 -- 172.21.15.97:0/2937253522 >> v1:172.21.15.97:6790/0 conn(0x7efd3c1430c0 legacy=0x7efd3c138300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:46.043 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:46.041+0000 7efd40b8f640 1 -- 172.21.15.97:0/2937253522 shutdown_connections 2024-03-25T18:23:46.043 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:46.041+0000 7efd40b8f640 1 -- 172.21.15.97:0/2937253522 >> 172.21.15.97:0/2937253522 conn(0x7efd3c0b7ea0 msgr2=0x7efd3c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:23:46.043 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:46.041+0000 7efd40b8f640 1 -- 172.21.15.97:0/2937253522 shutdown_connections 2024-03-25T18:23:46.043 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:46.041+0000 7efd40b8f640 1 -- 172.21.15.97:0/2937253522 wait complete. 2024-03-25T18:23:46.043 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:46.041+0000 7efd40b8f640 1 librados: shutdown 2024-03-25T18:23:46.055 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:23:52.861 DEBUG:teuthology.orchestra.run.smithi097:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:23:52.979 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:52.977+0000 7f1625686640 10 monclient: get_monmap_and_config 2024-03-25T18:23:52.979 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:52.977+0000 7f1625686640 10 monclient: build_initial_monmap 2024-03-25T18:23:52.979 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:52.977+0000 7f1625686640 10 monclient: monmap: 2024-03-25T18:23:52.979 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:23:52.979 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:23:52.979 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:23:52.978770+0000 2024-03-25T18:23:52.979 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:23:52.978770+0000 2024-03-25T18:23:52.979 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:23:52.979 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:23:52.979 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:23:52.979 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T18:23:52.979 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-b 2024-03-25T18:23:52.980 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:23:52.981 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:52.978+0000 7f1625686640 1 Processor -- start 2024-03-25T18:23:52.981 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:52.978+0000 7f1625686640 1 -- start start 2024-03-25T18:23:52.981 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:52.978+0000 7f1625686640 10 monclient: init 2024-03-25T18:23:52.981 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:52.979+0000 7f1625686640 10 monclient: _reopen_session rank -1 2024-03-25T18:23:52.981 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:52.979+0000 7f1625686640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:23:52.981 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:52.979+0000 7f1625686640 10 monclient(hunting): picked mon.noname-a con 0x7f16200a98f0 addr v1:172.21.15.97:6789/0 2024-03-25T18:23:52.981 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:52.979+0000 7f1625686640 10 monclient(hunting): picked mon.noname-c con 0x7f162013f710 addr v1:172.21.15.98:6789/0 2024-03-25T18:23:52.981 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:52.979+0000 7f1625686640 10 monclient(hunting): picked mon.noname-b con 0x7f16201430c0 addr v1:172.21.15.97:6790/0 2024-03-25T18:23:52.981 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:52.979+0000 7f1625686640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1620142520 con 0x7f16200a98f0 2024-03-25T18:23:52.981 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:52.979+0000 7f1625686640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16200b28a0 con 0x7f162013f710 2024-03-25T18:23:52.981 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:52.979+0000 7f1625686640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16200b2a30 con 0x7f16201430c0 2024-03-25T18:23:52.982 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:52.979+0000 7f1625686640 10 monclient(hunting): _renew_subs 2024-03-25T18:23:52.982 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:52.979+0000 7f1625686640 10 monclient(hunting): authenticate will time out at 8045.255162s 2024-03-25T18:23:52.983 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:52.982+0000 7f161f7fe640 1 --1- >> v1:172.21.15.97:6790/0 conn(0x7f16201430c0 0x7f162014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6790/0 says I am v1:172.21.15.97:58900/0 (socket says 172.21.15.97:58900) 2024-03-25T18:23:52.983 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:52.982+0000 7f161f7fe640 1 -- 172.21.15.97:0/2361151219 learned_addr learned my addr 172.21.15.97:0/2361151219 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:23:52.992 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:52.990+0000 7f161e7fc640 1 -- 172.21.15.97:0/2361151219 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 160903184 0 0) 0x7f16200b2a30 con 0x7f16201430c0 2024-03-25T18:23:52.992 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:52.990+0000 7f161e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:23:52.992 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:52.990+0000 7f161e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:52.992 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:52.990+0000 7f161e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:52.992 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:52.990+0000 7f161e7fc640 10 monclient(hunting): my global_id is 4556 2024-03-25T18:23:52.992 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:52.990+0000 7f161e7fc640 1 -- 172.21.15.97:0/2361151219 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f160c003520 con 0x7f16201430c0 2024-03-25T18:23:53.001 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:52.998+0000 7f161e7fc640 1 -- 172.21.15.97:0/2361151219 <== mon.2 v1:172.21.15.97:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3495595684 0 0) 0x7f160c003520 con 0x7f16201430c0 2024-03-25T18:23:53.001 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:52.998+0000 7f161e7fc640 1 -- 172.21.15.97:0/2361151219 >> v1:172.21.15.98:6789/0 conn(0x7f162013f710 legacy=0x7f16201429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:53.001 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:52.998+0000 7f161e7fc640 1 -- 172.21.15.97:0/2361151219 >> v1:172.21.15.97:6789/0 conn(0x7f16200a98f0 legacy=0x7f16200a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:53.001 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:52.998+0000 7f161e7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:23:53.001 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:52.998+0000 7f161e7fc640 1 monclient: found mon.noname-b 2024-03-25T18:23:53.001 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:52.998+0000 7f161e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:23:53.001 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:52.998+0000 7f161e7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.97:6790/0 2024-03-25T18:23:53.002 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:52.998+0000 7f161e7fc640 1 -- 172.21.15.97:0/2361151219 --> v1:172.21.15.97:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16200b44c0 con 0x7f16201430c0 2024-03-25T18:23:53.002 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:52.999+0000 7f161e7fc640 10 monclient: _finish_auth 0 2024-03-25T18:23:53.002 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:52.999+0000 7f161e7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:23:53.002 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:52.999+0000 7f161e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:23:53.002 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:52.999+0000 7f161e7fc640 1 -- 172.21.15.97:0/2361151219 <== mon.2 v1:172.21.15.97:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f1618001690 con 0x7f16201430c0 2024-03-25T18:23:53.002 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:52.999+0000 7f161e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:53.002 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:52.999+0000 7f161e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:23:53.002 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:52.999+0000 7f161e7fc640 10 monclient: dump: 2024-03-25T18:23:53.002 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:23:53.002 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:23:53.002 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:23:53.002 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:23:53.002 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:53.002 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:23:53.003 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:23:53.003 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:23:53.003 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:23:53.003 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:23:53.003 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:52.999+0000 7f1625686640 5 monclient: authenticate success, global_id 4556 2024-03-25T18:23:53.003 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:52.999+0000 7f1625686640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:23:53.007 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:53.005+0000 7f161e7fc640 1 -- 172.21.15.97:0/2361151219 <== mon.2 v1:172.21.15.97:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f16180017f0 con 0x7f16201430c0 2024-03-25T18:23:53.007 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:53.005+0000 7f161e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:23:53.007 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:53.005+0000 7f161e7fc640 1 -- 172.21.15.97:0/2361151219 <== mon.2 v1:172.21.15.97:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f1618001c40 con 0x7f16201430c0 2024-03-25T18:23:53.007 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:53.005+0000 7f1625686640 10 monclient: get_monmap_and_config success 2024-03-25T18:23:53.007 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:53.005+0000 7f1625686640 4 set_mon_vals no callback set 2024-03-25T18:23:53.007 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:53.005+0000 7f161e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:53.007 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:53.005+0000 7f161e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:23:53.007 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:53.005+0000 7f161e7fc640 10 monclient: dump: 2024-03-25T18:23:53.007 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:23:53.008 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:23:53.008 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:23:53.008 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:23:53.008 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:53.008 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:23:53.008 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:23:53.008 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:23:53.008 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:23:53.008 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:23:53.010 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:53.006+0000 7f1625686640 10 monclient: shutdown 2024-03-25T18:23:53.010 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:53.006+0000 7f1625686640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:23:53.010 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:53.006+0000 7f1625686640 1 -- 172.21.15.97:0/2361151219 >> v1:172.21.15.97:6790/0 conn(0x7f16201430c0 legacy=0x7f162014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:53.011 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:53.006+0000 7f1625686640 1 -- 172.21.15.97:0/2361151219 shutdown_connections 2024-03-25T18:23:53.011 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:53.006+0000 7f1625686640 1 -- 172.21.15.97:0/2361151219 >> 172.21.15.97:0/2361151219 conn(0x7f16200b7ea0 msgr2=0x7f16200b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:23:53.011 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:53.006+0000 7f1625686640 1 -- 172.21.15.97:0/2361151219 shutdown_connections 2024-03-25T18:23:53.011 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:53.006+0000 7f1625686640 1 -- 172.21.15.97:0/2361151219 wait complete. 2024-03-25T18:23:53.011 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:53.007+0000 7f1625686640 10 monclient: build_initial_monmap 2024-03-25T18:23:53.011 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:53.007+0000 7f1625686640 10 monclient: monmap: 2024-03-25T18:23:53.011 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:23:53.011 INFO:teuthology.misc.health.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:23:53.011 INFO:teuthology.misc.health.smithi097.stderr:last_changed 0.000000 2024-03-25T18:23:53.011 INFO:teuthology.misc.health.smithi097.stderr:created 0.000000 2024-03-25T18:23:53.011 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:23:53.011 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 1 2024-03-25T18:23:53.011 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:23:53.011 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T18:23:53.012 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-c 2024-03-25T18:23:53.012 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:23:53.012 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:53.007+0000 7f1625686640 1 librados: starting msgr at 2024-03-25T18:23:53.012 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:53.007+0000 7f1625686640 1 librados: starting objecter 2024-03-25T18:23:53.012 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:53.007+0000 7f1625686640 1 Processor -- start 2024-03-25T18:23:53.012 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:53.008+0000 7f1625686640 1 -- start start 2024-03-25T18:23:53.012 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:53.008+0000 7f1625686640 1 librados: setting wanted keys 2024-03-25T18:23:53.012 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:53.008+0000 7f1625686640 1 librados: calling monclient init 2024-03-25T18:23:53.012 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:53.008+0000 7f1625686640 10 monclient: init 2024-03-25T18:23:53.012 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:53.008+0000 7f1625686640 10 monclient: _reopen_session rank -1 2024-03-25T18:23:53.012 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:53.008+0000 7f1625686640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:23:53.012 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:53.008+0000 7f1625686640 10 monclient(hunting): picked mon.noname-a con 0x7f16200a98f0 addr v1:172.21.15.97:6789/0 2024-03-25T18:23:53.012 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:53.008+0000 7f1625686640 10 monclient(hunting): picked mon.noname-c con 0x7f162013f710 addr v1:172.21.15.97:6790/0 2024-03-25T18:23:53.012 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:53.008+0000 7f1625686640 10 monclient(hunting): picked mon.noname-b con 0x7f16201430c0 addr v1:172.21.15.98:6789/0 2024-03-25T18:23:53.013 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:53.008+0000 7f1625686640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16201d83a0 con 0x7f16200a98f0 2024-03-25T18:23:53.013 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:53.008+0000 7f1625686640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16200abe50 con 0x7f16201430c0 2024-03-25T18:23:53.013 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:53.008+0000 7f1625686640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1620138f30 con 0x7f162013f710 2024-03-25T18:23:53.013 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:53.008+0000 7f1625686640 10 monclient(hunting): _renew_subs 2024-03-25T18:23:53.013 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:53.008+0000 7f1625686640 10 monclient(hunting): authenticate will time out at 8045.284629s 2024-03-25T18:23:53.014 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:53.012+0000 7f161effd640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f16200a98f0 0x7f16201d7c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:55214/0 (socket says 172.21.15.97:55214) 2024-03-25T18:23:53.014 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:53.012+0000 7f161effd640 1 -- 172.21.15.97:0/2254983522 learned_addr learned my addr 172.21.15.97:0/2254983522 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:23:53.014 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:53.013+0000 7f1617fff640 10 client.?.objecter ms_handle_connect 0x7f16200a98f0 2024-03-25T18:23:53.014 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:53.013+0000 7f1617fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:53.025 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:53.023+0000 7f1617fff640 1 -- 172.21.15.97:0/2254983522 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 65019802 0 0) 0x7f16201d83a0 con 0x7f16200a98f0 2024-03-25T18:23:53.025 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:53.023+0000 7f1617fff640 10 monclient(hunting): _negotiate 2024-03-25T18:23:53.025 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:53.023+0000 7f1617fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:53.025 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:53.023+0000 7f1617fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:53.025 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:53.023+0000 7f1617fff640 10 monclient(hunting): my global_id is 4563 2024-03-25T18:23:53.025 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:53.023+0000 7f1617fff640 1 -- 172.21.15.97:0/2254983522 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1600004eb0 con 0x7f16200a98f0 2024-03-25T18:23:53.026 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:53.025+0000 7f1617fff640 10 client.?.objecter ms_handle_connect 0x7f162013f710 2024-03-25T18:23:53.026 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:53.025+0000 7f1617fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:53.036 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:53.035+0000 7f1617fff640 1 -- 172.21.15.97:0/2254983522 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2867955325 0 0) 0x7f1620138f30 con 0x7f162013f710 2024-03-25T18:23:53.037 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:53.035+0000 7f1617fff640 10 monclient(hunting): _negotiate 2024-03-25T18:23:53.037 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:53.035+0000 7f1617fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:53.037 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:53.035+0000 7f1617fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:53.037 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:53.035+0000 7f1617fff640 10 monclient(hunting): my global_id is 4559 2024-03-25T18:23:53.037 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:53.035+0000 7f1617fff640 1 -- 172.21.15.97:0/2254983522 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16201d83a0 con 0x7f162013f710 2024-03-25T18:23:53.046 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:53.044+0000 7f1617fff640 1 -- 172.21.15.97:0/2254983522 <== mon.2 v1:172.21.15.97:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 233665281 0 0) 0x7f16201d83a0 con 0x7f162013f710 2024-03-25T18:23:53.046 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:53.045+0000 7f1617fff640 1 -- 172.21.15.97:0/2254983522 --> v1:172.21.15.97:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1620138f30 con 0x7f162013f710 2024-03-25T18:23:53.046 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:53.045+0000 7f1617fff640 1 -- 172.21.15.97:0/2254983522 <== mon.2 v1:172.21.15.97:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f16080018a0 con 0x7f162013f710 2024-03-25T18:23:53.047 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:53.045+0000 7f1617fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:53.047 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:53.045+0000 7f1617fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:23:53.047 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:53.045+0000 7f1617fff640 10 monclient(hunting): dump: 2024-03-25T18:23:53.047 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:23:53.047 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:23:53.047 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:23:53.047 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:23:53.047 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:53.047 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:23:53.047 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:23:53.047 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:23:53.047 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:23:53.047 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:23:53.047 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:53.045+0000 7f1617fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:23:53.048 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:53.045+0000 7f1617fff640 10 client.?.objecter ms_handle_connect 0x7f16201430c0 2024-03-25T18:23:53.048 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:53.045+0000 7f1617fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:53.051 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:53.049+0000 7f1617fff640 1 -- 172.21.15.97:0/2254983522 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1304201211 0 0) 0x7f1600004eb0 con 0x7f16200a98f0 2024-03-25T18:23:53.051 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:53.049+0000 7f1617fff640 1 -- 172.21.15.97:0/2254983522 --> v1:172.21.15.97:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f16201d83a0 con 0x7f16200a98f0 2024-03-25T18:23:53.051 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:53.049+0000 7f1617fff640 1 -- 172.21.15.97:0/2254983522 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f1610001db0 con 0x7f16200a98f0 2024-03-25T18:23:53.051 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:53.049+0000 7f1617fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:53.051 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:53.049+0000 7f1617fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:23:53.051 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:53.049+0000 7f1617fff640 10 monclient(hunting): dump: 2024-03-25T18:23:53.051 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:23:53.051 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:23:53.051 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:23:53.051 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:23:53.051 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:53.051 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:23:53.052 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:23:53.052 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:23:53.052 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:23:53.052 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:23:53.059 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:53.057+0000 7f1617fff640 1 -- 172.21.15.97:0/2254983522 <== mon.2 v1:172.21.15.97:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3668328847 0 0) 0x7f1620138f30 con 0x7f162013f710 2024-03-25T18:23:53.059 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:53.057+0000 7f1617fff640 1 -- 172.21.15.97:0/2254983522 >> v1:172.21.15.98:6789/0 conn(0x7f16201430c0 legacy=0x7f1620133f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:53.060 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:53.057+0000 7f1617fff640 1 -- 172.21.15.97:0/2254983522 >> v1:172.21.15.97:6789/0 conn(0x7f16200a98f0 legacy=0x7f16201d7c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:53.060 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:53.057+0000 7f1617fff640 10 monclient: _finish_hunting 0 2024-03-25T18:23:53.060 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:53.057+0000 7f1617fff640 1 monclient: found mon.c 2024-03-25T18:23:53.060 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:53.057+0000 7f1617fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:23:53.060 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:53.057+0000 7f1617fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.97:6790/0 2024-03-25T18:23:53.060 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:53.057+0000 7f1617fff640 1 -- 172.21.15.97:0/2254983522 --> v1:172.21.15.97:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16201d7490 con 0x7f162013f710 2024-03-25T18:23:53.060 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:53.057+0000 7f1617fff640 10 monclient: _finish_auth 0 2024-03-25T18:23:53.060 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:53.057+0000 7f1617fff640 10 monclient: _check_auth_tickets 2024-03-25T18:23:53.060 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:53.057+0000 7f1617fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:23:53.060 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:53.058+0000 7f1625686640 5 monclient: authenticate success, global_id 4559 2024-03-25T18:23:53.060 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:53.058+0000 7f1625686640 10 monclient: _renew_subs 2024-03-25T18:23:53.060 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:53.058+0000 7f1625686640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.97:6790/0 2024-03-25T18:23:53.060 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:53.058+0000 7f1625686640 1 -- 172.21.15.97:0/2254983522 --> v1:172.21.15.97:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16201d7770 con 0x7f162013f710 2024-03-25T18:23:53.060 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:53.058+0000 7f1625686640 10 client.4559.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:23:53.060 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:53.058+0000 7f1625686640 10 monclient: _renew_subs 2024-03-25T18:23:53.061 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:53.058+0000 7f1625686640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.97:6790/0 2024-03-25T18:23:53.061 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:53.058+0000 7f1625686640 1 -- 172.21.15.97:0/2254983522 --> v1:172.21.15.97:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1620138240 con 0x7f162013f710 2024-03-25T18:23:53.061 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:53.058+0000 7f1625686640 1 librados: init done 2024-03-25T18:23:53.061 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:53.059+0000 7f1625686640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:53.061 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:53.059+0000 7f1625686640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:53.061 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:53.059+0000 7f1625686640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.97:6790/0 2024-03-25T18:23:53.061 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:53.059+0000 7f1625686640 1 -- 172.21.15.97:0/2254983522 --> v1:172.21.15.97:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16200627b0 con 0x7f162013f710 2024-03-25T18:23:53.341 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:53.340+0000 7f1617fff640 1 -- 172.21.15.97:0/2254983522 <== mon.2 v1:172.21.15.97:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f16080035f0 con 0x7f162013f710 2024-03-25T18:23:53.341 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:53.340+0000 7f1617fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:23:53.341 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:53.340+0000 7f161dffb640 4 set_mon_vals no callback set 2024-03-25T18:23:53.354 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:53.353+0000 7f1617fff640 1 -- 172.21.15.97:0/2254983522 <== mon.2 v1:172.21.15.97:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f16080039c0 con 0x7f162013f710 2024-03-25T18:23:53.354 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:53.353+0000 7f1617fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:53.354 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:53.353+0000 7f1617fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:23:53.355 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:53.353+0000 7f1617fff640 10 monclient: dump: 2024-03-25T18:23:53.355 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:23:53.355 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:23:53.355 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:23:53.355 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:23:53.355 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:53.355 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:23:53.355 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:23:53.355 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:23:53.355 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:23:53.355 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:23:53.356 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:53.354+0000 7f1617fff640 1 -- 172.21.15.97:0/2254983522 <== mon.2 v1:172.21.15.97:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2377622137 0 0) 0x7f1608003ba0 con 0x7f162013f710 2024-03-25T18:23:53.356 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:53.354+0000 7f161effd640 10 monclient: get_auth_request con 0x7f160006de80 auth_method 0 2024-03-25T18:23:53.356 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:53.355+0000 7f1617fff640 10 client.4559.objecter ms_handle_connect 0x7f160006de80 2024-03-25T18:23:53.361 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:53.359+0000 7f1617fff640 1 -- 172.21.15.97:0/2254983522 <== mon.2 v1:172.21.15.97:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 430711918 0 0) 0x7f16080535a0 con 0x7f162013f710 2024-03-25T18:23:53.361 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:53.359+0000 7f1617fff640 10 client.4559.objecter ms_dispatch 0x7f1620000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:23:53.361 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:53.359+0000 7f1617fff640 3 client.4559.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:23:53.361 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:53.359+0000 7f1617fff640 3 client.4559.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:23:53.361 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:53.360+0000 7f1617fff640 20 client.4559.objecter dump_active .. 0 homeless 2024-03-25T18:23:53.361 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:53.360+0000 7f1617fff640 1 -- 172.21.15.97:0/2254983522 <== mon.2 v1:172.21.15.97:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f160804f0b0 con 0x7f162013f710 2024-03-25T18:23:53.362 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:53.360+0000 7f1617fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:53.362 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:53.360+0000 7f1617fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:23:53.446 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:07:55.484 34931== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:53.453 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:07:55.492 34931== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:53.487 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:07:55.525 34931== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:53.509 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:07:55.553 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:53.513 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:07:55.556 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:53.521 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:07:55.559 34935== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:53.534 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:07:55.577 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:53.534 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:07:55.573 34935== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:53.555 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:07:55.594 34935== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:53.578 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:53.576+0000 7f1625686640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:53.578 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:53.576+0000 7f1625686640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:53.578 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:53.576+0000 7f1625686640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.97:6790/0 2024-03-25T18:23:53.578 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:53.576+0000 7f1625686640 1 -- 172.21.15.97:0/2254983522 --> v1:172.21.15.97:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f16201d7900 con 0x7f162013f710 2024-03-25T18:23:53.760 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:53.759+0000 7f1617fff640 1 -- 172.21.15.97:0/2254983522 <== mon.2 v1:172.21.15.97:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2169427412) 0x7f1608050fe0 con 0x7f162013f710 2024-03-25T18:23:53.761 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:53.759+0000 7f1617fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:53.761 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:53.759+0000 7f1617fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:23:53.761 INFO:teuthology.misc.health.smithi097.stdout: 2024-03-25T18:23:53.761 INFO:teuthology.misc.health.smithi097.stdout:{"epoch":10,"fsid":"4c3d472c-79b8-486b-9e3d-2738d4306754","created":"2024-03-25T18:16:14.737309+0000","modified":"2024-03-25T18:16:20.357492+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:20.357492+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cb5152b9-262f-45c3-94b8-18d0e81df558","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6daf8793-0838-473d-a7d6-9a0d2894c75d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"136f1493-3b9c-4f43-a2e7-641482d69dae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"397d538c-4f64-4daf-9e8b-662d09f91327","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4f1df08-663f-4bda-93f8-3878ab7c24b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c4b18d70-a3ce-4b15-9b70-eae5525bd761","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8dd4033a-de0e-4f8a-96dc-59f3c1f825b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0077722e-ffa6-4a3d-9891-929d9869b286","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:23:53.763 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:53.761+0000 7f1625686640 10 librados: watch_flush enter 2024-03-25T18:23:53.763 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:53.761+0000 7f1625686640 10 librados: watch_flush exit 2024-03-25T18:23:53.763 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:53.761+0000 7f1625686640 20 client.4559.objecter shutdown clearing up homeless session... 2024-03-25T18:23:53.763 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:53.761+0000 7f1625686640 10 client.4559.objecter successfully canceled tick 2024-03-25T18:23:53.763 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:53.761+0000 7f1625686640 1 -- 172.21.15.97:0/2254983522 >> v1:172.21.15.98:6800/800387573 conn(0x7f160006de80 legacy=0x7f1600070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:53.764 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:53.761+0000 7f1625686640 10 monclient: shutdown 2024-03-25T18:23:53.764 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:53.761+0000 7f1625686640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:23:53.764 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:53.761+0000 7f1625686640 1 -- 172.21.15.97:0/2254983522 >> v1:172.21.15.97:6790/0 conn(0x7f162013f710 legacy=0x7f1620138820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:53.764 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:53.761+0000 7f1625686640 1 -- 172.21.15.97:0/2254983522 shutdown_connections 2024-03-25T18:23:53.764 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:53.761+0000 7f1625686640 1 -- 172.21.15.97:0/2254983522 >> 172.21.15.97:0/2254983522 conn(0x7f16200b7ea0 msgr2=0x7f1620074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:23:53.764 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:53.761+0000 7f1625686640 1 -- 172.21.15.97:0/2254983522 shutdown_connections 2024-03-25T18:23:53.764 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:53.761+0000 7f1625686640 1 -- 172.21.15.97:0/2254983522 wait complete. 2024-03-25T18:23:53.764 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:23:53.761+0000 7f1625686640 1 librados: shutdown 2024-03-25T18:23:53.776 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:24:00.583 DEBUG:teuthology.orchestra.run.smithi097:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:24:00.700 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:00.699+0000 7f09755eb640 10 monclient: get_monmap_and_config 2024-03-25T18:24:00.700 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:00.699+0000 7f09755eb640 10 monclient: build_initial_monmap 2024-03-25T18:24:00.700 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:00.699+0000 7f09755eb640 10 monclient: monmap: 2024-03-25T18:24:00.700 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:24:00.700 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:24:00.700 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:24:00.700326+0000 2024-03-25T18:24:00.700 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:24:00.700326+0000 2024-03-25T18:24:00.701 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:00.701 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:24:00.701 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:24:00.701 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T18:24:00.701 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-b 2024-03-25T18:24:00.701 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:24:00.702 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:00.699+0000 7f09755eb640 1 Processor -- start 2024-03-25T18:24:00.702 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:00.699+0000 7f09755eb640 1 -- start start 2024-03-25T18:24:00.702 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:00.699+0000 7f09755eb640 10 monclient: init 2024-03-25T18:24:00.702 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:00.700+0000 7f09755eb640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:00.702 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:00.700+0000 7f09755eb640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:24:00.702 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:00.700+0000 7f09755eb640 10 monclient(hunting): picked mon.noname-b con 0x7f09700a98f0 addr v1:172.21.15.97:6790/0 2024-03-25T18:24:00.702 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:00.700+0000 7f09755eb640 10 monclient(hunting): picked mon.noname-a con 0x7f097013f710 addr v1:172.21.15.97:6789/0 2024-03-25T18:24:00.702 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:00.700+0000 7f09755eb640 10 monclient(hunting): picked mon.noname-c con 0x7f09701430c0 addr v1:172.21.15.98:6789/0 2024-03-25T18:24:00.702 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:00.700+0000 7f09755eb640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0970142520 con 0x7f097013f710 2024-03-25T18:24:00.702 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:00.700+0000 7f09755eb640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09700b28a0 con 0x7f09701430c0 2024-03-25T18:24:00.703 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:00.700+0000 7f09755eb640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09700b2a30 con 0x7f09700a98f0 2024-03-25T18:24:00.703 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:00.700+0000 7f09755eb640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:00.703 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:00.700+0000 7f09755eb640 10 monclient(hunting): authenticate will time out at 8052.976405s 2024-03-25T18:24:00.715 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:00.713+0000 7f096effd640 1 --1- >> v1:172.21.15.97:6790/0 conn(0x7f09700a98f0 0x7f09700a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6790/0 says I am v1:172.21.15.97:33810/0 (socket says 172.21.15.97:33810) 2024-03-25T18:24:00.715 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:00.713+0000 7f096effd640 1 -- 172.21.15.97:0/2709545260 learned_addr learned my addr 172.21.15.97:0/2709545260 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:24:00.723 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:00.721+0000 7f096e7fc640 1 -- 172.21.15.97:0/2709545260 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4126886824 0 0) 0x7f09700b2a30 con 0x7f09700a98f0 2024-03-25T18:24:00.723 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:00.721+0000 7f096e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:24:00.723 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:00.721+0000 7f096e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:00.723 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:00.721+0000 7f096e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:00.723 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:00.721+0000 7f096e7fc640 10 monclient(hunting): my global_id is 4562 2024-03-25T18:24:00.723 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:00.721+0000 7f096e7fc640 1 -- 172.21.15.97:0/2709545260 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f095c003520 con 0x7f09700a98f0 2024-03-25T18:24:00.727 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:00.726+0000 7f096e7fc640 1 -- 172.21.15.97:0/2709545260 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 833350227 0 0) 0x7f09700b28a0 con 0x7f09701430c0 2024-03-25T18:24:00.727 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:00.726+0000 7f096e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:24:00.727 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:00.726+0000 7f096e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:00.727 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:00.726+0000 7f096e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:00.727 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:00.726+0000 7f096e7fc640 10 monclient(hunting): my global_id is 4573 2024-03-25T18:24:00.727 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:00.726+0000 7f096e7fc640 1 -- 172.21.15.97:0/2709545260 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09700b2a30 con 0x7f09701430c0 2024-03-25T18:24:00.732 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:00.730+0000 7f096e7fc640 1 -- 172.21.15.97:0/2709545260 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1247282759 0 0) 0x7f0970142520 con 0x7f097013f710 2024-03-25T18:24:00.732 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:00.730+0000 7f096e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:24:00.732 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:00.730+0000 7f096e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:00.732 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:00.730+0000 7f096e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:00.732 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:00.730+0000 7f096e7fc640 10 monclient(hunting): my global_id is 4566 2024-03-25T18:24:00.732 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:00.730+0000 7f096e7fc640 1 -- 172.21.15.97:0/2709545260 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09700b28a0 con 0x7f097013f710 2024-03-25T18:24:00.744 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:00.742+0000 7f096e7fc640 1 -- 172.21.15.97:0/2709545260 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4193403941 0 0) 0x7f09700b2a30 con 0x7f09701430c0 2024-03-25T18:24:00.744 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:00.742+0000 7f096e7fc640 1 -- 172.21.15.97:0/2709545260 >> v1:172.21.15.97:6790/0 conn(0x7f09700a98f0 legacy=0x7f09700a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:00.744 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:00.742+0000 7f096e7fc640 1 -- 172.21.15.97:0/2709545260 >> v1:172.21.15.97:6789/0 conn(0x7f097013f710 legacy=0x7f09701429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:00.744 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:00.742+0000 7f096e7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:24:00.744 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:00.742+0000 7f096e7fc640 1 monclient: found mon.noname-c 2024-03-25T18:24:00.744 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:00.742+0000 7f096e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:00.744 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:00.742+0000 7f096e7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.98:6789/0 2024-03-25T18:24:00.744 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:00.742+0000 7f096e7fc640 1 -- 172.21.15.97:0/2709545260 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09700b44c0 con 0x7f09701430c0 2024-03-25T18:24:00.744 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:00.742+0000 7f096e7fc640 10 monclient: _finish_auth 0 2024-03-25T18:24:00.744 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:00.742+0000 7f096e7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:24:00.744 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:00.742+0000 7f096e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:00.745 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:00.742+0000 7f096e7fc640 1 -- 172.21.15.97:0/2709545260 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f09680019f0 con 0x7f09701430c0 2024-03-25T18:24:00.745 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:00.742+0000 7f096e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:00.745 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:00.742+0000 7f096e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:24:00.745 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:00.742+0000 7f096e7fc640 10 monclient: dump: 2024-03-25T18:24:00.745 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:24:00.745 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:24:00.745 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:24:00.745 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:24:00.745 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:00.745 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:24:00.745 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:24:00.745 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:24:00.745 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:24:00.745 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:24:00.745 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:00.742+0000 7f09755eb640 5 monclient: authenticate success, global_id 4573 2024-03-25T18:24:00.745 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:00.742+0000 7f09755eb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:24:00.751 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:00.750+0000 7f096e7fc640 1 -- 172.21.15.97:0/2709545260 <== mon.1 v1:172.21.15.98:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0968001b50 con 0x7f09701430c0 2024-03-25T18:24:00.751 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:00.750+0000 7f096e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:00.751 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:00.750+0000 7f096e7fc640 1 -- 172.21.15.97:0/2709545260 <== mon.1 v1:172.21.15.98:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f0968001f20 con 0x7f09701430c0 2024-03-25T18:24:00.751 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:00.750+0000 7f09755eb640 10 monclient: get_monmap_and_config success 2024-03-25T18:24:00.751 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:00.750+0000 7f09755eb640 4 set_mon_vals no callback set 2024-03-25T18:24:00.752 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:00.750+0000 7f096e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:00.752 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:00.750+0000 7f096e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:24:00.752 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:00.750+0000 7f096e7fc640 10 monclient: dump: 2024-03-25T18:24:00.752 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:24:00.752 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:24:00.752 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:24:00.752 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:24:00.752 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:00.752 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:24:00.752 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:24:00.752 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:24:00.752 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:24:00.752 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:24:00.755 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:00.750+0000 7f09755eb640 10 monclient: shutdown 2024-03-25T18:24:00.755 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:00.750+0000 7f09755eb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:00.755 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:00.750+0000 7f09755eb640 1 -- 172.21.15.97:0/2709545260 >> v1:172.21.15.98:6789/0 conn(0x7f09701430c0 legacy=0x7f097014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:00.755 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:00.751+0000 7f09755eb640 1 -- 172.21.15.97:0/2709545260 shutdown_connections 2024-03-25T18:24:00.755 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:00.751+0000 7f09755eb640 1 -- 172.21.15.97:0/2709545260 >> 172.21.15.97:0/2709545260 conn(0x7f09700b7ea0 msgr2=0x7f09700b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:00.755 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:00.751+0000 7f09755eb640 1 -- 172.21.15.97:0/2709545260 shutdown_connections 2024-03-25T18:24:00.755 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:00.751+0000 7f09755eb640 1 -- 172.21.15.97:0/2709545260 wait complete. 2024-03-25T18:24:00.755 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:00.751+0000 7f09755eb640 10 monclient: build_initial_monmap 2024-03-25T18:24:00.755 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:00.751+0000 7f09755eb640 10 monclient: monmap: 2024-03-25T18:24:00.755 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:24:00.755 INFO:teuthology.misc.health.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:24:00.755 INFO:teuthology.misc.health.smithi097.stderr:last_changed 0.000000 2024-03-25T18:24:00.755 INFO:teuthology.misc.health.smithi097.stderr:created 0.000000 2024-03-25T18:24:00.756 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:00.756 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 1 2024-03-25T18:24:00.756 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:24:00.756 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T18:24:00.756 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-c 2024-03-25T18:24:00.756 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:24:00.756 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:00.752+0000 7f09755eb640 1 librados: starting msgr at 2024-03-25T18:24:00.756 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:00.752+0000 7f09755eb640 1 librados: starting objecter 2024-03-25T18:24:00.756 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:00.752+0000 7f09755eb640 1 Processor -- start 2024-03-25T18:24:00.756 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:00.752+0000 7f09755eb640 1 -- start start 2024-03-25T18:24:00.756 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:00.752+0000 7f09755eb640 1 librados: setting wanted keys 2024-03-25T18:24:00.756 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:00.752+0000 7f09755eb640 1 librados: calling monclient init 2024-03-25T18:24:00.756 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:00.752+0000 7f09755eb640 10 monclient: init 2024-03-25T18:24:00.756 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:00.753+0000 7f09755eb640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:00.756 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:00.753+0000 7f09755eb640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:24:00.757 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:00.753+0000 7f09755eb640 10 monclient(hunting): picked mon.noname-b con 0x7f09700a98f0 addr v1:172.21.15.98:6789/0 2024-03-25T18:24:00.757 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:00.753+0000 7f09755eb640 10 monclient(hunting): picked mon.noname-c con 0x7f097013f710 addr v1:172.21.15.97:6790/0 2024-03-25T18:24:00.757 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:00.753+0000 7f09755eb640 10 monclient(hunting): picked mon.noname-a con 0x7f09701430c0 addr v1:172.21.15.97:6789/0 2024-03-25T18:24:00.757 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:00.753+0000 7f09755eb640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f097013c190 con 0x7f09701430c0 2024-03-25T18:24:00.757 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:00.753+0000 7f09755eb640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09700b7cf0 con 0x7f09700a98f0 2024-03-25T18:24:00.757 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:00.753+0000 7f09755eb640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f097013ad50 con 0x7f097013f710 2024-03-25T18:24:00.757 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:00.753+0000 7f09755eb640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:00.757 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:00.753+0000 7f09755eb640 10 monclient(hunting): authenticate will time out at 8053.029559s 2024-03-25T18:24:00.766 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:00.765+0000 7f096effd640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f09700a98f0 0x7f097013ba80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.97:45472/0 (socket says 172.21.15.97:45472) 2024-03-25T18:24:00.766 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:00.765+0000 7f096effd640 1 -- 172.21.15.97:0/3973371419 learned_addr learned my addr 172.21.15.97:0/3973371419 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:24:00.767 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:00.765+0000 7f09677fe640 10 client.?.objecter ms_handle_connect 0x7f09700a98f0 2024-03-25T18:24:00.767 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:00.765+0000 7f09677fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:00.770 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:00.769+0000 7f09677fe640 10 client.?.objecter ms_handle_connect 0x7f097013f710 2024-03-25T18:24:00.770 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:00.769+0000 7f09677fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:00.772 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:00.771+0000 7f09677fe640 1 -- 172.21.15.97:0/3973371419 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1622905213 0 0) 0x7f09700b7cf0 con 0x7f09700a98f0 2024-03-25T18:24:00.772 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:00.771+0000 7f09677fe640 10 monclient(hunting): _negotiate 2024-03-25T18:24:00.772 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:00.771+0000 7f09677fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:00.772 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:00.771+0000 7f09677fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:00.772 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:00.771+0000 7f09677fe640 10 monclient(hunting): my global_id is 4576 2024-03-25T18:24:00.772 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:00.771+0000 7f09677fe640 1 -- 172.21.15.97:0/3973371419 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0950004eb0 con 0x7f09700a98f0 2024-03-25T18:24:00.774 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:00.772+0000 7f09677fe640 10 client.?.objecter ms_handle_connect 0x7f09701430c0 2024-03-25T18:24:00.774 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:00.772+0000 7f09677fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:00.786 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:00.784+0000 7f09677fe640 1 -- 172.21.15.97:0/3973371419 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 364994713 0 0) 0x7f0950004eb0 con 0x7f09700a98f0 2024-03-25T18:24:00.786 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:00.784+0000 7f09677fe640 1 -- 172.21.15.97:0/3973371419 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f09700b7cf0 con 0x7f09700a98f0 2024-03-25T18:24:00.786 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:00.784+0000 7f09677fe640 1 -- 172.21.15.97:0/3973371419 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f0960001950 con 0x7f09700a98f0 2024-03-25T18:24:00.786 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:00.784+0000 7f09677fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:00.786 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:00.784+0000 7f09677fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:24:00.786 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:00.784+0000 7f09677fe640 10 monclient(hunting): dump: 2024-03-25T18:24:00.786 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:24:00.786 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:24:00.786 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:24:00.787 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:24:00.787 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:00.787 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:24:00.787 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:24:00.787 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:24:00.787 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:24:00.787 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:24:00.787 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:00.784+0000 7f09677fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:24:00.787 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:00.784+0000 7f09677fe640 1 -- 172.21.15.97:0/3973371419 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1175126468 0 0) 0x7f09700b44c0 con 0x7f09701430c0 2024-03-25T18:24:00.787 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:00.784+0000 7f09677fe640 10 monclient(hunting): _negotiate 2024-03-25T18:24:00.787 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:00.784+0000 7f09677fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:00.787 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:00.784+0000 7f09677fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:00.787 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:00.784+0000 7f09677fe640 10 monclient(hunting): my global_id is 4569 2024-03-25T18:24:00.787 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:00.784+0000 7f09677fe640 1 -- 172.21.15.97:0/3973371419 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0950004eb0 con 0x7f09701430c0 2024-03-25T18:24:00.791 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:00.790+0000 7f09677fe640 1 -- 172.21.15.97:0/3973371419 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1363875705 0 0) 0x7f097013ad50 con 0x7f097013f710 2024-03-25T18:24:00.791 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:00.790+0000 7f09677fe640 10 monclient(hunting): _negotiate 2024-03-25T18:24:00.792 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:00.790+0000 7f09677fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:00.792 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:00.790+0000 7f09677fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:00.792 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:00.790+0000 7f09677fe640 10 monclient(hunting): my global_id is 4565 2024-03-25T18:24:00.792 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:00.790+0000 7f09677fe640 1 -- 172.21.15.97:0/3973371419 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09700b44c0 con 0x7f097013f710 2024-03-25T18:24:00.794 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:00.792+0000 7f09677fe640 1 -- 172.21.15.97:0/3973371419 <== mon.1 v1:172.21.15.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2823749169 0 0) 0x7f09700b7cf0 con 0x7f09700a98f0 2024-03-25T18:24:00.794 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:00.792+0000 7f09677fe640 1 -- 172.21.15.97:0/3973371419 >> v1:172.21.15.97:6790/0 conn(0x7f097013f710 legacy=0x7f09701df270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:00.794 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:00.792+0000 7f09677fe640 1 -- 172.21.15.97:0/3973371419 >> v1:172.21.15.97:6789/0 conn(0x7f09701430c0 legacy=0x7f09701e29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:00.794 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:00.792+0000 7f09677fe640 10 monclient: _finish_hunting 0 2024-03-25T18:24:00.795 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:00.792+0000 7f09677fe640 1 monclient: found mon.b 2024-03-25T18:24:00.795 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:00.792+0000 7f09677fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:00.795 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:00.792+0000 7f09677fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:24:00.795 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:00.792+0000 7f09677fe640 1 -- 172.21.15.97:0/3973371419 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f097013afc0 con 0x7f09700a98f0 2024-03-25T18:24:00.795 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:00.792+0000 7f09677fe640 10 monclient: _finish_auth 0 2024-03-25T18:24:00.795 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:00.792+0000 7f09677fe640 10 monclient: _check_auth_tickets 2024-03-25T18:24:00.795 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:00.792+0000 7f09677fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:00.795 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:00.792+0000 7f09755eb640 5 monclient: authenticate success, global_id 4576 2024-03-25T18:24:00.795 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:00.792+0000 7f09755eb640 10 monclient: _renew_subs 2024-03-25T18:24:00.795 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:00.792+0000 7f09755eb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:24:00.795 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:00.792+0000 7f09755eb640 1 -- 172.21.15.97:0/3973371419 --> v1:172.21.15.98:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f097013b2a0 con 0x7f09700a98f0 2024-03-25T18:24:00.795 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:00.792+0000 7f09755eb640 10 client.4576.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:24:00.795 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:00.792+0000 7f09755eb640 10 monclient: _renew_subs 2024-03-25T18:24:00.795 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:00.792+0000 7f09755eb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:24:00.795 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:00.792+0000 7f09755eb640 1 -- 172.21.15.97:0/3973371419 --> v1:172.21.15.98:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f097012fae0 con 0x7f09700a98f0 2024-03-25T18:24:00.796 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:00.792+0000 7f09755eb640 1 librados: init done 2024-03-25T18:24:00.796 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:00.793+0000 7f09755eb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:00.796 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:00.793+0000 7f09755eb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:00.796 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:00.793+0000 7f09755eb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:24:00.796 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:00.793+0000 7f09755eb640 1 -- 172.21.15.97:0/3973371419 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09700b3ef0 con 0x7f09700a98f0 2024-03-25T18:24:01.061 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:01.060+0000 7f09677fe640 1 -- 172.21.15.97:0/3973371419 <== mon.1 v1:172.21.15.98:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f09600047e0 con 0x7f09700a98f0 2024-03-25T18:24:01.062 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:01.060+0000 7f09677fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:01.062 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:01.060+0000 7f09677fe640 1 -- 172.21.15.97:0/3973371419 <== mon.1 v1:172.21.15.98:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f0960004b70 con 0x7f09700a98f0 2024-03-25T18:24:01.062 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:01.060+0000 7f09677fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:01.062 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:01.060+0000 7f09677fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:24:01.062 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:01.060+0000 7f09677fe640 10 monclient: dump: 2024-03-25T18:24:01.062 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:24:01.062 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:24:01.062 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:24:01.062 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:24:01.062 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:01.062 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:24:01.062 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:24:01.062 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:24:01.063 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:24:01.063 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:24:01.063 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:01.060+0000 7f096dffb640 4 set_mon_vals no callback set 2024-03-25T18:24:01.063 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:01.061+0000 7f09677fe640 1 -- 172.21.15.97:0/3973371419 <== mon.1 v1:172.21.15.98:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2377622137 0 0) 0x7f0960004d90 con 0x7f09700a98f0 2024-03-25T18:24:01.063 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:01.062+0000 7f0967fff640 10 monclient: get_auth_request con 0x7f095006de80 auth_method 0 2024-03-25T18:24:01.063 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:01.062+0000 7f09677fe640 10 client.4576.objecter ms_handle_connect 0x7f095006de80 2024-03-25T18:24:01.069 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:01.067+0000 7f09677fe640 1 -- 172.21.15.97:0/3973371419 <== mon.1 v1:172.21.15.98:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 430711918 0 0) 0x7f0960054850 con 0x7f09700a98f0 2024-03-25T18:24:01.069 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:01.067+0000 7f09677fe640 10 client.4576.objecter ms_dispatch 0x7f0970000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:24:01.069 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:01.067+0000 7f09677fe640 3 client.4576.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:24:01.069 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:01.067+0000 7f09677fe640 3 client.4576.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:24:01.070 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:01.068+0000 7f09677fe640 20 client.4576.objecter dump_active .. 0 homeless 2024-03-25T18:24:01.071 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:01.069+0000 7f09677fe640 1 -- 172.21.15.97:0/3973371419 <== mon.1 v1:172.21.15.98:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0960050360 con 0x7f09700a98f0 2024-03-25T18:24:01.071 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:01.069+0000 7f09677fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:01.071 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:01.069+0000 7f09677fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:24:01.296 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:01.294+0000 7f09755eb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:01.296 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:01.294+0000 7f09755eb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:01.296 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:01.294+0000 7f09755eb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:24:01.296 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:01.294+0000 7f09755eb640 1 -- 172.21.15.97:0/3973371419 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f09700627b0 con 0x7f09700a98f0 2024-03-25T18:24:01.317 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:01.315+0000 7f09677fe640 1 -- 172.21.15.97:0/3973371419 <== mon.1 v1:172.21.15.98:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2169427412) 0x7f0960052290 con 0x7f09700a98f0 2024-03-25T18:24:01.317 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:01.315+0000 7f09677fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:01.317 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:01.315+0000 7f09677fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:24:01.317 INFO:teuthology.misc.health.smithi097.stdout: 2024-03-25T18:24:01.317 INFO:teuthology.misc.health.smithi097.stdout:{"epoch":10,"fsid":"4c3d472c-79b8-486b-9e3d-2738d4306754","created":"2024-03-25T18:16:14.737309+0000","modified":"2024-03-25T18:16:20.357492+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:20.357492+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cb5152b9-262f-45c3-94b8-18d0e81df558","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6daf8793-0838-473d-a7d6-9a0d2894c75d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"136f1493-3b9c-4f43-a2e7-641482d69dae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"397d538c-4f64-4daf-9e8b-662d09f91327","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4f1df08-663f-4bda-93f8-3878ab7c24b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c4b18d70-a3ce-4b15-9b70-eae5525bd761","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8dd4033a-de0e-4f8a-96dc-59f3c1f825b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0077722e-ffa6-4a3d-9891-929d9869b286","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:24:01.319 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:01.317+0000 7f09755eb640 10 librados: watch_flush enter 2024-03-25T18:24:01.320 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:01.317+0000 7f09755eb640 10 librados: watch_flush exit 2024-03-25T18:24:01.320 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:01.317+0000 7f09755eb640 20 client.4576.objecter shutdown clearing up homeless session... 2024-03-25T18:24:01.320 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:01.317+0000 7f09755eb640 10 client.4576.objecter successfully canceled tick 2024-03-25T18:24:01.320 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:01.317+0000 7f09755eb640 1 -- 172.21.15.97:0/3973371419 >> v1:172.21.15.98:6800/800387573 conn(0x7f095006de80 legacy=0x7f0950070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:01.320 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:01.318+0000 7f09755eb640 10 monclient: shutdown 2024-03-25T18:24:01.320 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:01.318+0000 7f09755eb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:01.320 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:01.318+0000 7f09755eb640 1 -- 172.21.15.97:0/3973371419 >> v1:172.21.15.98:6789/0 conn(0x7f09700a98f0 legacy=0x7f097013ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:01.320 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:01.318+0000 7f09755eb640 1 -- 172.21.15.97:0/3973371419 shutdown_connections 2024-03-25T18:24:01.320 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:01.318+0000 7f09755eb640 1 -- 172.21.15.97:0/3973371419 >> 172.21.15.97:0/3973371419 conn(0x7f09700b7ea0 msgr2=0x7f0970074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:01.320 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:01.318+0000 7f09755eb640 1 -- 172.21.15.97:0/3973371419 shutdown_connections 2024-03-25T18:24:01.320 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:01.318+0000 7f09755eb640 1 -- 172.21.15.97:0/3973371419 wait complete. 2024-03-25T18:24:01.320 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:01.318+0000 7f09755eb640 1 librados: shutdown 2024-03-25T18:24:01.331 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:24:08.137 DEBUG:teuthology.orchestra.run.smithi097:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:24:08.254 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.252+0000 7fb9ad281640 10 monclient: get_monmap_and_config 2024-03-25T18:24:08.254 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.252+0000 7fb9ad281640 10 monclient: build_initial_monmap 2024-03-25T18:24:08.254 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.252+0000 7fb9ad281640 10 monclient: monmap: 2024-03-25T18:24:08.254 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:24:08.254 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:24:08.254 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:24:08.254161+0000 2024-03-25T18:24:08.254 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:24:08.254161+0000 2024-03-25T18:24:08.254 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:08.255 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:24:08.255 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:24:08.255 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T18:24:08.255 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-b 2024-03-25T18:24:08.255 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:24:08.256 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.253+0000 7fb9ad281640 1 Processor -- start 2024-03-25T18:24:08.256 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.253+0000 7fb9ad281640 1 -- start start 2024-03-25T18:24:08.256 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.253+0000 7fb9ad281640 10 monclient: init 2024-03-25T18:24:08.256 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.254+0000 7fb9ad281640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:08.256 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.254+0000 7fb9ad281640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:24:08.256 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.254+0000 7fb9ad281640 10 monclient(hunting): picked mon.noname-c con 0x7fb9a80a98f0 addr v1:172.21.15.98:6789/0 2024-03-25T18:24:08.256 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.254+0000 7fb9ad281640 10 monclient(hunting): picked mon.noname-b con 0x7fb9a813f710 addr v1:172.21.15.97:6790/0 2024-03-25T18:24:08.256 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.254+0000 7fb9ad281640 10 monclient(hunting): picked mon.noname-a con 0x7fb9a81430c0 addr v1:172.21.15.97:6789/0 2024-03-25T18:24:08.256 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.254+0000 7fb9ad281640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9a8142520 con 0x7fb9a81430c0 2024-03-25T18:24:08.257 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.254+0000 7fb9ad281640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9a80b28a0 con 0x7fb9a80a98f0 2024-03-25T18:24:08.257 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.254+0000 7fb9ad281640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9a80b2a30 con 0x7fb9a813f710 2024-03-25T18:24:08.257 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.254+0000 7fb9ad281640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:08.257 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.254+0000 7fb9ad281640 10 monclient(hunting): authenticate will time out at 8060.530567s 2024-03-25T18:24:08.267 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.266+0000 7fb9a6ffd640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fb9a80a98f0 0x7fb9a80a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.97:50110/0 (socket says 172.21.15.97:50110) 2024-03-25T18:24:08.267 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.266+0000 7fb9a6ffd640 1 -- 172.21.15.97:0/1011347051 learned_addr learned my addr 172.21.15.97:0/1011347051 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:24:08.275 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.274+0000 7fb9a5ffb640 1 -- 172.21.15.97:0/1011347051 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3508805076 0 0) 0x7fb9a80b28a0 con 0x7fb9a80a98f0 2024-03-25T18:24:08.276 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.274+0000 7fb9a5ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:24:08.276 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.274+0000 7fb9a5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:08.276 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.274+0000 7fb9a5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:08.276 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.274+0000 7fb9a5ffb640 10 monclient(hunting): my global_id is 4579 2024-03-25T18:24:08.276 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.274+0000 7fb9a5ffb640 1 -- 172.21.15.97:0/1011347051 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb990003520 con 0x7fb9a80a98f0 2024-03-25T18:24:08.292 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.290+0000 7fb9a5ffb640 1 -- 172.21.15.97:0/1011347051 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2291543327 0 0) 0x7fb990003520 con 0x7fb9a80a98f0 2024-03-25T18:24:08.292 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.290+0000 7fb9a5ffb640 1 -- 172.21.15.97:0/1011347051 >> v1:172.21.15.97:6790/0 conn(0x7fb9a813f710 legacy=0x7fb9a81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:08.292 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.290+0000 7fb9a5ffb640 1 -- 172.21.15.97:0/1011347051 >> v1:172.21.15.97:6789/0 conn(0x7fb9a81430c0 legacy=0x7fb9a814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:08.292 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.290+0000 7fb9a5ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:24:08.292 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.290+0000 7fb9a5ffb640 1 monclient: found mon.noname-c 2024-03-25T18:24:08.292 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.290+0000 7fb9a5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:08.292 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.290+0000 7fb9a5ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.98:6789/0 2024-03-25T18:24:08.292 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.290+0000 7fb9a5ffb640 1 -- 172.21.15.97:0/1011347051 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9a80b44c0 con 0x7fb9a80a98f0 2024-03-25T18:24:08.292 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.290+0000 7fb9a5ffb640 10 monclient: _finish_auth 0 2024-03-25T18:24:08.292 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.290+0000 7fb9a5ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:24:08.292 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.290+0000 7fb9a5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:08.292 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.290+0000 7fb9a5ffb640 1 -- 172.21.15.97:0/1011347051 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7fb998001690 con 0x7fb9a80a98f0 2024-03-25T18:24:08.292 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.290+0000 7fb9ad281640 5 monclient: authenticate success, global_id 4579 2024-03-25T18:24:08.293 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.290+0000 7fb9ad281640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:24:08.293 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.290+0000 7fb9a5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:08.293 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.290+0000 7fb9a5ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:24:08.293 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.290+0000 7fb9a5ffb640 10 monclient: dump: 2024-03-25T18:24:08.293 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:24:08.293 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:24:08.293 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:24:08.293 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:24:08.293 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:08.293 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:24:08.293 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:24:08.293 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:24:08.293 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:24:08.293 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:24:08.293 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.290+0000 7fb9ad281640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:24:08.295 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.293+0000 7fb9a5ffb640 1 -- 172.21.15.97:0/1011347051 <== mon.1 v1:172.21.15.98:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb998001870 con 0x7fb9a80a98f0 2024-03-25T18:24:08.295 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.293+0000 7fb9a5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:08.295 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.293+0000 7fb9a5ffb640 1 -- 172.21.15.97:0/1011347051 <== mon.1 v1:172.21.15.98:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7fb998001c80 con 0x7fb9a80a98f0 2024-03-25T18:24:08.295 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.293+0000 7fb9a5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:08.295 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.293+0000 7fb9a5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:24:08.295 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.293+0000 7fb9a5ffb640 10 monclient: dump: 2024-03-25T18:24:08.295 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:24:08.295 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:24:08.295 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:24:08.296 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:24:08.296 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:08.296 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:24:08.296 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:24:08.296 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:24:08.296 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:24:08.296 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:24:08.296 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.293+0000 7fb9ad281640 10 monclient: get_monmap_and_config success 2024-03-25T18:24:08.296 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.293+0000 7fb9ad281640 4 set_mon_vals no callback set 2024-03-25T18:24:08.298 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.294+0000 7fb9ad281640 10 monclient: shutdown 2024-03-25T18:24:08.298 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.294+0000 7fb9ad281640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:08.298 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.294+0000 7fb9ad281640 1 -- 172.21.15.97:0/1011347051 >> v1:172.21.15.98:6789/0 conn(0x7fb9a80a98f0 legacy=0x7fb9a80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:08.299 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.294+0000 7fb9ad281640 1 -- 172.21.15.97:0/1011347051 shutdown_connections 2024-03-25T18:24:08.299 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.294+0000 7fb9ad281640 1 -- 172.21.15.97:0/1011347051 >> 172.21.15.97:0/1011347051 conn(0x7fb9a80b7ea0 msgr2=0x7fb9a80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:08.299 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.294+0000 7fb9ad281640 1 -- 172.21.15.97:0/1011347051 shutdown_connections 2024-03-25T18:24:08.299 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.294+0000 7fb9ad281640 1 -- 172.21.15.97:0/1011347051 wait complete. 2024-03-25T18:24:08.299 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.295+0000 7fb9ad281640 10 monclient: build_initial_monmap 2024-03-25T18:24:08.299 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.295+0000 7fb9ad281640 10 monclient: monmap: 2024-03-25T18:24:08.299 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:24:08.299 INFO:teuthology.misc.health.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:24:08.299 INFO:teuthology.misc.health.smithi097.stderr:last_changed 0.000000 2024-03-25T18:24:08.299 INFO:teuthology.misc.health.smithi097.stderr:created 0.000000 2024-03-25T18:24:08.299 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:08.299 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 1 2024-03-25T18:24:08.299 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:24:08.299 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T18:24:08.300 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-c 2024-03-25T18:24:08.300 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:24:08.300 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.295+0000 7fb9ad281640 1 librados: starting msgr at 2024-03-25T18:24:08.300 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.295+0000 7fb9ad281640 1 librados: starting objecter 2024-03-25T18:24:08.300 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.295+0000 7fb9ad281640 1 Processor -- start 2024-03-25T18:24:08.300 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.295+0000 7fb9ad281640 1 -- start start 2024-03-25T18:24:08.300 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.295+0000 7fb9ad281640 1 librados: setting wanted keys 2024-03-25T18:24:08.300 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.295+0000 7fb9ad281640 1 librados: calling monclient init 2024-03-25T18:24:08.300 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.295+0000 7fb9ad281640 10 monclient: init 2024-03-25T18:24:08.300 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.296+0000 7fb9ad281640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:08.300 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.296+0000 7fb9ad281640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:24:08.300 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.296+0000 7fb9ad281640 10 monclient(hunting): picked mon.noname-c con 0x7fb9a80a98f0 addr v1:172.21.15.97:6790/0 2024-03-25T18:24:08.300 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.296+0000 7fb9ad281640 10 monclient(hunting): picked mon.noname-a con 0x7fb9a813f710 addr v1:172.21.15.97:6789/0 2024-03-25T18:24:08.300 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.296+0000 7fb9ad281640 10 monclient(hunting): picked mon.noname-b con 0x7fb9a81430c0 addr v1:172.21.15.98:6789/0 2024-03-25T18:24:08.300 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.296+0000 7fb9ad281640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9a8138b20 con 0x7fb9a813f710 2024-03-25T18:24:08.301 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.296+0000 7fb9ad281640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9a8137a80 con 0x7fb9a81430c0 2024-03-25T18:24:08.301 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.296+0000 7fb9ad281640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9a8139880 con 0x7fb9a80a98f0 2024-03-25T18:24:08.301 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.296+0000 7fb9ad281640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:08.301 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.296+0000 7fb9ad281640 10 monclient(hunting): authenticate will time out at 8060.572373s 2024-03-25T18:24:08.304 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.303+0000 7fb9a67fc640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fb9a813f710 0x7fb9a8139170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:42838/0 (socket says 172.21.15.97:42838) 2024-03-25T18:24:08.304 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.303+0000 7fb9a67fc640 1 -- 172.21.15.97:0/1167855498 learned_addr learned my addr 172.21.15.97:0/1167855498 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:24:08.307 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.305+0000 7fb98f7fe640 10 client.?.objecter ms_handle_connect 0x7fb9a813f710 2024-03-25T18:24:08.307 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.305+0000 7fb98f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:08.311 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.309+0000 7fb98f7fe640 10 client.?.objecter ms_handle_connect 0x7fb9a80a98f0 2024-03-25T18:24:08.311 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.309+0000 7fb98f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:08.312 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.311+0000 7fb98f7fe640 10 client.?.objecter ms_handle_connect 0x7fb9a81430c0 2024-03-25T18:24:08.312 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.311+0000 7fb98f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:08.317 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.316+0000 7fb98f7fe640 1 -- 172.21.15.97:0/1167855498 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3682738933 0 0) 0x7fb9a8137a80 con 0x7fb9a81430c0 2024-03-25T18:24:08.318 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.316+0000 7fb98f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:24:08.318 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.316+0000 7fb98f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:08.318 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.316+0000 7fb98f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:08.318 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.316+0000 7fb98f7fe640 10 monclient(hunting): my global_id is 4582 2024-03-25T18:24:08.318 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.316+0000 7fb98f7fe640 1 -- 172.21.15.97:0/1167855498 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb994004eb0 con 0x7fb9a81430c0 2024-03-25T18:24:08.321 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.320+0000 7fb98f7fe640 1 -- 172.21.15.97:0/1167855498 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2661326606 0 0) 0x7fb9a8138b20 con 0x7fb9a813f710 2024-03-25T18:24:08.321 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.320+0000 7fb98f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:24:08.322 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.320+0000 7fb98f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:08.322 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.320+0000 7fb98f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:08.322 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.320+0000 7fb98f7fe640 10 monclient(hunting): my global_id is 4575 2024-03-25T18:24:08.322 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.320+0000 7fb98f7fe640 1 -- 172.21.15.97:0/1167855498 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9a8137a80 con 0x7fb9a813f710 2024-03-25T18:24:08.332 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.330+0000 7fb98f7fe640 1 -- 172.21.15.97:0/1167855498 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 254069803 0 0) 0x7fb994004eb0 con 0x7fb9a81430c0 2024-03-25T18:24:08.332 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.330+0000 7fb98f7fe640 1 -- 172.21.15.97:0/1167855498 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb9a8138b20 con 0x7fb9a81430c0 2024-03-25T18:24:08.332 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.330+0000 7fb98f7fe640 1 -- 172.21.15.97:0/1167855498 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7fb9a0001b20 con 0x7fb9a81430c0 2024-03-25T18:24:08.332 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.330+0000 7fb98f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:08.332 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.330+0000 7fb98f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:24:08.332 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.330+0000 7fb98f7fe640 10 monclient(hunting): dump: 2024-03-25T18:24:08.332 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:24:08.332 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:24:08.333 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:24:08.333 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:24:08.333 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:08.333 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:24:08.333 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:24:08.333 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:24:08.333 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:24:08.333 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:24:08.333 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.330+0000 7fb98f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:24:08.334 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.332+0000 7fb98f7fe640 1 -- 172.21.15.97:0/1167855498 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 215500512 0 0) 0x7fb9a8139880 con 0x7fb9a80a98f0 2024-03-25T18:24:08.334 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.332+0000 7fb98f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:24:08.334 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.332+0000 7fb98f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:08.334 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.332+0000 7fb98f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:08.334 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.332+0000 7fb98f7fe640 10 monclient(hunting): my global_id is 4571 2024-03-25T18:24:08.334 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.332+0000 7fb98f7fe640 1 -- 172.21.15.97:0/1167855498 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb994004eb0 con 0x7fb9a80a98f0 2024-03-25T18:24:08.334 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.332+0000 7fb98f7fe640 1 -- 172.21.15.97:0/1167855498 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 467143811 0 0) 0x7fb9a8137a80 con 0x7fb9a813f710 2024-03-25T18:24:08.334 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.332+0000 7fb98f7fe640 1 -- 172.21.15.97:0/1167855498 --> v1:172.21.15.97:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb9a8139880 con 0x7fb9a813f710 2024-03-25T18:24:08.334 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.332+0000 7fb98f7fe640 1 -- 172.21.15.97:0/1167855498 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7fb99c001930 con 0x7fb9a813f710 2024-03-25T18:24:08.335 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.332+0000 7fb98f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:08.335 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.332+0000 7fb98f7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:24:08.335 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.332+0000 7fb98f7fe640 10 monclient(hunting): dump: 2024-03-25T18:24:08.335 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:24:08.335 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:24:08.335 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:24:08.335 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:24:08.335 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:08.335 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:24:08.335 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:24:08.335 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:24:08.335 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:24:08.335 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:24:08.340 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.338+0000 7fb98f7fe640 1 -- 172.21.15.97:0/1167855498 <== mon.1 v1:172.21.15.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 39135704 0 0) 0x7fb9a8138b20 con 0x7fb9a81430c0 2024-03-25T18:24:08.340 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.338+0000 7fb98f7fe640 1 -- 172.21.15.97:0/1167855498 >> v1:172.21.15.97:6790/0 conn(0x7fb9a80a98f0 legacy=0x7fb9a8137370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:08.340 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.338+0000 7fb98f7fe640 1 -- 172.21.15.97:0/1167855498 >> v1:172.21.15.97:6789/0 conn(0x7fb9a813f710 legacy=0x7fb9a8139170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:08.340 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.338+0000 7fb98f7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:24:08.340 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.338+0000 7fb98f7fe640 1 monclient: found mon.b 2024-03-25T18:24:08.340 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.338+0000 7fb98f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:08.341 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.338+0000 7fb98f7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:24:08.341 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.338+0000 7fb98f7fe640 1 -- 172.21.15.97:0/1167855498 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9a8139a10 con 0x7fb9a81430c0 2024-03-25T18:24:08.341 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.338+0000 7fb98f7fe640 10 monclient: _finish_auth 0 2024-03-25T18:24:08.341 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.338+0000 7fb98f7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:24:08.341 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.338+0000 7fb98f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:08.341 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.338+0000 7fb9ad281640 5 monclient: authenticate success, global_id 4582 2024-03-25T18:24:08.341 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.338+0000 7fb9ad281640 10 monclient: _renew_subs 2024-03-25T18:24:08.341 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.338+0000 7fb9ad281640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:24:08.341 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.338+0000 7fb9ad281640 1 -- 172.21.15.97:0/1167855498 --> v1:172.21.15.98:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9a81dab00 con 0x7fb9a81430c0 2024-03-25T18:24:08.341 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.338+0000 7fb9ad281640 10 client.4582.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:24:08.341 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.338+0000 7fb9ad281640 10 monclient: _renew_subs 2024-03-25T18:24:08.341 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.338+0000 7fb9ad281640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:24:08.341 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.338+0000 7fb9ad281640 1 -- 172.21.15.97:0/1167855498 --> v1:172.21.15.98:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9a813b4f0 con 0x7fb9a81430c0 2024-03-25T18:24:08.341 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.338+0000 7fb9ad281640 1 librados: init done 2024-03-25T18:24:08.342 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.339+0000 7fb9ad281640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:08.342 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.339+0000 7fb9ad281640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:08.342 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.339+0000 7fb9ad281640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:24:08.342 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.339+0000 7fb9ad281640 1 -- 172.21.15.97:0/1167855498 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9a80b3e80 con 0x7fb9a81430c0 2024-03-25T18:24:08.611 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.609+0000 7fb98f7fe640 1 -- 172.21.15.97:0/1167855498 <== mon.1 v1:172.21.15.98:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb9a0002030 con 0x7fb9a81430c0 2024-03-25T18:24:08.611 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.609+0000 7fb98f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:08.611 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.609+0000 7fb98f7fe640 1 -- 172.21.15.97:0/1167855498 <== mon.1 v1:172.21.15.98:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7fb9a0003b10 con 0x7fb9a81430c0 2024-03-25T18:24:08.611 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.609+0000 7fb98f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:08.611 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.609+0000 7fb98f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:24:08.611 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.609+0000 7fb98f7fe640 10 monclient: dump: 2024-03-25T18:24:08.611 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:24:08.611 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:24:08.611 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:24:08.611 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:24:08.611 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:08.612 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:24:08.612 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:24:08.612 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:24:08.612 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:24:08.612 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:24:08.612 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.609+0000 7fb9a57fa640 4 set_mon_vals no callback set 2024-03-25T18:24:08.612 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.610+0000 7fb98f7fe640 1 -- 172.21.15.97:0/1167855498 <== mon.1 v1:172.21.15.98:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2377622137 0 0) 0x7fb9a0019ae0 con 0x7fb9a81430c0 2024-03-25T18:24:08.612 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.610+0000 7fb9a6ffd640 10 monclient: get_auth_request con 0x7fb99406df20 auth_method 0 2024-03-25T18:24:08.612 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.611+0000 7fb98f7fe640 10 client.4582.objecter ms_handle_connect 0x7fb99406df20 2024-03-25T18:24:08.616 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.614+0000 7fb98f7fe640 1 -- 172.21.15.97:0/1167855498 <== mon.1 v1:172.21.15.98:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 430711918 0 0) 0x7fb9a0053880 con 0x7fb9a81430c0 2024-03-25T18:24:08.616 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.614+0000 7fb98f7fe640 10 client.4582.objecter ms_dispatch 0x7fb9a8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:24:08.616 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.614+0000 7fb98f7fe640 3 client.4582.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:24:08.616 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.614+0000 7fb98f7fe640 3 client.4582.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:24:08.616 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.614+0000 7fb98f7fe640 20 client.4582.objecter dump_active .. 0 homeless 2024-03-25T18:24:08.616 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.614+0000 7fb98f7fe640 1 -- 172.21.15.97:0/1167855498 <== mon.1 v1:172.21.15.98:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb9a004f1c0 con 0x7fb9a81430c0 2024-03-25T18:24:08.616 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.614+0000 7fb98f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:08.616 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.614+0000 7fb98f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:24:08.852 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.851+0000 7fb9ad281640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:08.852 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.851+0000 7fb9ad281640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:08.852 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.851+0000 7fb9ad281640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:24:08.852 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.851+0000 7fb9ad281640 1 -- 172.21.15.97:0/1167855498 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb9a80627b0 con 0x7fb9a81430c0 2024-03-25T18:24:08.884 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.882+0000 7fb98f7fe640 1 -- 172.21.15.97:0/1167855498 <== mon.1 v1:172.21.15.98:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2169427412) 0x7fb9a00512c0 con 0x7fb9a81430c0 2024-03-25T18:24:08.884 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.882+0000 7fb98f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:08.884 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.882+0000 7fb98f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:24:08.884 INFO:teuthology.misc.health.smithi097.stdout: 2024-03-25T18:24:08.884 INFO:teuthology.misc.health.smithi097.stdout:{"epoch":10,"fsid":"4c3d472c-79b8-486b-9e3d-2738d4306754","created":"2024-03-25T18:16:14.737309+0000","modified":"2024-03-25T18:16:20.357492+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:20.357492+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cb5152b9-262f-45c3-94b8-18d0e81df558","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6daf8793-0838-473d-a7d6-9a0d2894c75d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"136f1493-3b9c-4f43-a2e7-641482d69dae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"397d538c-4f64-4daf-9e8b-662d09f91327","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4f1df08-663f-4bda-93f8-3878ab7c24b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c4b18d70-a3ce-4b15-9b70-eae5525bd761","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8dd4033a-de0e-4f8a-96dc-59f3c1f825b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0077722e-ffa6-4a3d-9891-929d9869b286","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:24:08.887 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.885+0000 7fb9ad281640 10 librados: watch_flush enter 2024-03-25T18:24:08.887 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.885+0000 7fb9ad281640 10 librados: watch_flush exit 2024-03-25T18:24:08.887 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.885+0000 7fb9ad281640 20 client.4582.objecter shutdown clearing up homeless session... 2024-03-25T18:24:08.887 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.885+0000 7fb9ad281640 10 client.4582.objecter successfully canceled tick 2024-03-25T18:24:08.887 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.885+0000 7fb9ad281640 1 -- 172.21.15.97:0/1167855498 >> v1:172.21.15.98:6800/800387573 conn(0x7fb99406df20 legacy=0x7fb9940703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:08.887 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.885+0000 7fb9ad281640 10 monclient: shutdown 2024-03-25T18:24:08.887 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.885+0000 7fb9ad281640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:08.887 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.885+0000 7fb9ad281640 1 -- 172.21.15.97:0/1167855498 >> v1:172.21.15.98:6789/0 conn(0x7fb9a81430c0 legacy=0x7fb9a81e2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:08.887 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.885+0000 7fb9ad281640 1 -- 172.21.15.97:0/1167855498 shutdown_connections 2024-03-25T18:24:08.887 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.885+0000 7fb9ad281640 1 -- 172.21.15.97:0/1167855498 >> 172.21.15.97:0/1167855498 conn(0x7fb9a80b7ea0 msgr2=0x7fb9a8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:08.887 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.885+0000 7fb9ad281640 1 -- 172.21.15.97:0/1167855498 shutdown_connections 2024-03-25T18:24:08.888 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.885+0000 7fb9ad281640 1 -- 172.21.15.97:0/1167855498 wait complete. 2024-03-25T18:24:08.888 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:08.885+0000 7fb9ad281640 1 librados: shutdown 2024-03-25T18:24:08.899 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:24:13.320 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:08:15.359 34931== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:13.380 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:08:15.419 34931== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:13.414 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:08:15.453 34931== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:13.432 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:08:15.475 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:13.439 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:08:15.482 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:13.443 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:08:15.482 34935== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:13.451 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:08:15.489 34935== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:13.458 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:08:15.501 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:13.484 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:08:15.523 34935== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:15.706 DEBUG:teuthology.orchestra.run.smithi097:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:24:15.819 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:15.819+0000 7f4990789640 10 monclient: get_monmap_and_config 2024-03-25T18:24:15.819 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:15.819+0000 7f4990789640 10 monclient: build_initial_monmap 2024-03-25T18:24:15.819 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:15.819+0000 7f4990789640 10 monclient: monmap: 2024-03-25T18:24:15.820 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:24:15.820 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:24:15.820 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:24:15.819392+0000 2024-03-25T18:24:15.820 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:24:15.819392+0000 2024-03-25T18:24:15.820 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:15.820 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:24:15.820 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:24:15.820 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T18:24:15.820 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-b 2024-03-25T18:24:15.820 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:24:15.822 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:15.819+0000 7f4990789640 1 Processor -- start 2024-03-25T18:24:15.822 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:15.819+0000 7f4990789640 1 -- start start 2024-03-25T18:24:15.822 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:15.819+0000 7f4990789640 10 monclient: init 2024-03-25T18:24:15.822 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:15.820+0000 7f4990789640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:15.822 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:15.820+0000 7f4990789640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:24:15.822 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:15.820+0000 7f4990789640 10 monclient(hunting): picked mon.noname-b con 0x7f49880a98f0 addr v1:172.21.15.97:6790/0 2024-03-25T18:24:15.822 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:15.820+0000 7f4990789640 10 monclient(hunting): picked mon.noname-a con 0x7f498813f710 addr v1:172.21.15.97:6789/0 2024-03-25T18:24:15.822 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:15.820+0000 7f4990789640 10 monclient(hunting): picked mon.noname-c con 0x7f49881430c0 addr v1:172.21.15.98:6789/0 2024-03-25T18:24:15.822 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:15.820+0000 7f4990789640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4988142520 con 0x7f498813f710 2024-03-25T18:24:15.822 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:15.820+0000 7f4990789640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49880b28a0 con 0x7f49881430c0 2024-03-25T18:24:15.822 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:15.820+0000 7f4990789640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49880b2a30 con 0x7f49880a98f0 2024-03-25T18:24:15.822 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:15.820+0000 7f4990789640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:15.822 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:15.820+0000 7f4990789640 10 monclient(hunting): authenticate will time out at 8068.095742s 2024-03-25T18:24:15.827 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:15.826+0000 7f498dd24640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f498813f710 0x7f49881429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:50964/0 (socket says 172.21.15.97:50964) 2024-03-25T18:24:15.827 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:15.826+0000 7f498dd24640 1 -- 172.21.15.97:0/1921465316 learned_addr learned my addr 172.21.15.97:0/1921465316 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:24:15.839 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:15.838+0000 7f498d523640 1 -- 172.21.15.97:0/1921465316 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4066293027 0 0) 0x7f4988142520 con 0x7f498813f710 2024-03-25T18:24:15.839 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:15.838+0000 7f498d523640 10 monclient(hunting): _negotiate 2024-03-25T18:24:15.839 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:15.838+0000 7f498d523640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:15.839 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:15.838+0000 7f498d523640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:15.839 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:15.838+0000 7f498d523640 10 monclient(hunting): my global_id is 4578 2024-03-25T18:24:15.839 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:15.838+0000 7f498d523640 1 -- 172.21.15.97:0/1921465316 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4974003520 con 0x7f498813f710 2024-03-25T18:24:15.844 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:15.843+0000 7f498d523640 1 -- 172.21.15.97:0/1921465316 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3018156436 0 0) 0x7f49880b28a0 con 0x7f49881430c0 2024-03-25T18:24:15.844 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:15.843+0000 7f498d523640 10 monclient(hunting): _negotiate 2024-03-25T18:24:15.844 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:15.843+0000 7f498d523640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:15.844 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:15.843+0000 7f498d523640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:15.844 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:15.843+0000 7f498d523640 10 monclient(hunting): my global_id is 4585 2024-03-25T18:24:15.844 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:15.843+0000 7f498d523640 1 -- 172.21.15.97:0/1921465316 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4988142520 con 0x7f49881430c0 2024-03-25T18:24:15.858 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:15.857+0000 7f498d523640 1 -- 172.21.15.97:0/1921465316 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1831993908 0 0) 0x7f4974003520 con 0x7f498813f710 2024-03-25T18:24:15.858 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:15.857+0000 7f498d523640 1 -- 172.21.15.97:0/1921465316 >> v1:172.21.15.97:6790/0 conn(0x7f49880a98f0 legacy=0x7f49880a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:15.858 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:15.857+0000 7f498d523640 1 -- 172.21.15.97:0/1921465316 >> v1:172.21.15.98:6789/0 conn(0x7f49881430c0 legacy=0x7f498814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:15.858 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:15.857+0000 7f498d523640 10 monclient: _finish_hunting 0 2024-03-25T18:24:15.858 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:15.857+0000 7f498d523640 1 monclient: found mon.noname-a 2024-03-25T18:24:15.858 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:15.857+0000 7f498d523640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:15.858 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:15.857+0000 7f498d523640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.97:6789/0 2024-03-25T18:24:15.858 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:15.857+0000 7f498d523640 1 -- 172.21.15.97:0/1921465316 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49880b44c0 con 0x7f498813f710 2024-03-25T18:24:15.859 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:15.857+0000 7f498d523640 10 monclient: _finish_auth 0 2024-03-25T18:24:15.859 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:15.857+0000 7f498d523640 10 monclient: _check_auth_tickets 2024-03-25T18:24:15.859 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:15.857+0000 7f498d523640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:15.859 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:15.857+0000 7f498d523640 1 -- 172.21.15.97:0/1921465316 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f4978001690 con 0x7f498813f710 2024-03-25T18:24:15.859 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:15.857+0000 7f4990789640 5 monclient: authenticate success, global_id 4578 2024-03-25T18:24:15.859 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:15.857+0000 7f4990789640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:24:15.859 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:15.857+0000 7f498d523640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:15.859 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:15.857+0000 7f498d523640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:24:15.859 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:15.857+0000 7f498d523640 10 monclient: dump: 2024-03-25T18:24:15.859 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:24:15.859 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:24:15.859 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:24:15.859 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:24:15.859 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:15.859 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:24:15.860 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:24:15.860 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:24:15.860 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:24:15.860 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:24:15.860 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:15.857+0000 7f4990789640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:24:15.869 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:15.869+0000 7f498d523640 1 -- 172.21.15.97:0/1921465316 <== mon.0 v1:172.21.15.97:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f49780017f0 con 0x7f498813f710 2024-03-25T18:24:15.870 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:15.869+0000 7f498d523640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:15.870 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:15.869+0000 7f498d523640 1 -- 172.21.15.97:0/1921465316 <== mon.0 v1:172.21.15.97:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f4978001c00 con 0x7f498813f710 2024-03-25T18:24:15.870 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:15.869+0000 7f4990789640 10 monclient: get_monmap_and_config success 2024-03-25T18:24:15.870 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:15.869+0000 7f4990789640 4 set_mon_vals no callback set 2024-03-25T18:24:15.870 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:15.869+0000 7f498d523640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:15.870 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:15.869+0000 7f498d523640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:24:15.870 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:15.869+0000 7f498d523640 10 monclient: dump: 2024-03-25T18:24:15.870 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:24:15.870 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:24:15.870 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:24:15.871 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:24:15.871 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:15.871 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:24:15.871 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:24:15.871 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:24:15.871 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:24:15.871 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:24:15.873 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:15.870+0000 7f4990789640 10 monclient: shutdown 2024-03-25T18:24:15.873 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:15.870+0000 7f4990789640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:15.873 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:15.870+0000 7f4990789640 1 -- 172.21.15.97:0/1921465316 >> v1:172.21.15.97:6789/0 conn(0x7f498813f710 legacy=0x7f49881429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:15.873 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:15.870+0000 7f4990789640 1 -- 172.21.15.97:0/1921465316 shutdown_connections 2024-03-25T18:24:15.873 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:15.870+0000 7f4990789640 1 -- 172.21.15.97:0/1921465316 >> 172.21.15.97:0/1921465316 conn(0x7f49880b7ea0 msgr2=0x7f49880b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:15.873 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:15.870+0000 7f4990789640 1 -- 172.21.15.97:0/1921465316 shutdown_connections 2024-03-25T18:24:15.873 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:15.870+0000 7f4990789640 1 -- 172.21.15.97:0/1921465316 wait complete. 2024-03-25T18:24:15.873 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:15.871+0000 7f4990789640 10 monclient: build_initial_monmap 2024-03-25T18:24:15.874 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:15.871+0000 7f4990789640 10 monclient: monmap: 2024-03-25T18:24:15.874 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:24:15.874 INFO:teuthology.misc.health.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:24:15.874 INFO:teuthology.misc.health.smithi097.stderr:last_changed 0.000000 2024-03-25T18:24:15.874 INFO:teuthology.misc.health.smithi097.stderr:created 0.000000 2024-03-25T18:24:15.874 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:15.874 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 1 2024-03-25T18:24:15.874 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:24:15.874 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T18:24:15.874 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-c 2024-03-25T18:24:15.874 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:24:15.874 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:15.871+0000 7f4990789640 1 librados: starting msgr at 2024-03-25T18:24:15.874 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:15.871+0000 7f4990789640 1 librados: starting objecter 2024-03-25T18:24:15.874 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:15.871+0000 7f4990789640 1 Processor -- start 2024-03-25T18:24:15.874 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:15.871+0000 7f4990789640 1 -- start start 2024-03-25T18:24:15.874 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:15.871+0000 7f4990789640 1 librados: setting wanted keys 2024-03-25T18:24:15.875 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:15.871+0000 7f4990789640 1 librados: calling monclient init 2024-03-25T18:24:15.875 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:15.871+0000 7f4990789640 10 monclient: init 2024-03-25T18:24:15.875 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:15.872+0000 7f4990789640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:15.875 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:15.872+0000 7f4990789640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:24:15.875 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:15.872+0000 7f4990789640 10 monclient(hunting): picked mon.noname-b con 0x7f49880a98f0 addr v1:172.21.15.98:6789/0 2024-03-25T18:24:15.875 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:15.872+0000 7f4990789640 10 monclient(hunting): picked mon.noname-c con 0x7f498813f710 addr v1:172.21.15.97:6790/0 2024-03-25T18:24:15.875 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:15.872+0000 7f4990789640 10 monclient(hunting): picked mon.noname-a con 0x7f49881430c0 addr v1:172.21.15.97:6789/0 2024-03-25T18:24:15.875 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:15.872+0000 7f4990789640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4988137010 con 0x7f49881430c0 2024-03-25T18:24:15.875 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:15.872+0000 7f4990789640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4988135100 con 0x7f49880a98f0 2024-03-25T18:24:15.875 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:15.872+0000 7f4990789640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4988135290 con 0x7f498813f710 2024-03-25T18:24:15.875 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:15.872+0000 7f4990789640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:15.875 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:15.872+0000 7f4990789640 10 monclient(hunting): authenticate will time out at 8068.147414s 2024-03-25T18:24:15.887 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:15.886+0000 7f498dd24640 1 --1- >> v1:172.21.15.97:6790/0 conn(0x7f498813f710 0x7f49881dab00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.97:6790/0 says I am v1:172.21.15.97:49424/0 (socket says 172.21.15.97:49424) 2024-03-25T18:24:15.887 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:15.886+0000 7f498dd24640 1 -- 172.21.15.97:0/3589627776 learned_addr learned my addr 172.21.15.97:0/3589627776 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:24:15.887 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:15.887+0000 7f497effd640 10 client.?.objecter ms_handle_connect 0x7f498813f710 2024-03-25T18:24:15.887 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:15.887+0000 7f497effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:15.898 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:15.898+0000 7f497effd640 10 client.?.objecter ms_handle_connect 0x7f49881430c0 2024-03-25T18:24:15.898 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:15.898+0000 7f497effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:15.907 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:15.906+0000 7f497effd640 10 client.?.objecter ms_handle_connect 0x7f49880a98f0 2024-03-25T18:24:15.907 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:15.906+0000 7f497effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:15.907 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:15.906+0000 7f497effd640 1 -- 172.21.15.97:0/3589627776 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3533445450 0 0) 0x7f4988135290 con 0x7f498813f710 2024-03-25T18:24:15.907 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:15.906+0000 7f497effd640 10 monclient(hunting): _negotiate 2024-03-25T18:24:15.907 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:15.906+0000 7f497effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:15.907 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:15.906+0000 7f497effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:15.907 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:15.906+0000 7f497effd640 10 monclient(hunting): my global_id is 4577 2024-03-25T18:24:15.907 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:15.906+0000 7f497effd640 1 -- 172.21.15.97:0/3589627776 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f496c004eb0 con 0x7f498813f710 2024-03-25T18:24:15.925 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:15.925+0000 7f497effd640 1 -- 172.21.15.97:0/3589627776 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2830222691 0 0) 0x7f4988135100 con 0x7f49880a98f0 2024-03-25T18:24:15.925 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:15.925+0000 7f497effd640 10 monclient(hunting): _negotiate 2024-03-25T18:24:15.925 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:15.925+0000 7f497effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:15.925 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:15.925+0000 7f497effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:15.925 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:15.925+0000 7f497effd640 10 monclient(hunting): my global_id is 4588 2024-03-25T18:24:15.925 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:15.925+0000 7f497effd640 1 -- 172.21.15.97:0/3589627776 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4988135290 con 0x7f49880a98f0 2024-03-25T18:24:15.926 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:15.926+0000 7f497effd640 1 -- 172.21.15.97:0/3589627776 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4245521833 0 0) 0x7f4988137010 con 0x7f49881430c0 2024-03-25T18:24:15.926 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:15.926+0000 7f497effd640 10 monclient(hunting): _negotiate 2024-03-25T18:24:15.926 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:15.926+0000 7f497effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:15.926 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:15.926+0000 7f497effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:15.927 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:15.926+0000 7f497effd640 10 monclient(hunting): my global_id is 4581 2024-03-25T18:24:15.927 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:15.926+0000 7f497effd640 1 -- 172.21.15.97:0/3589627776 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4988135100 con 0x7f49881430c0 2024-03-25T18:24:15.928 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:15.927+0000 7f497effd640 1 -- 172.21.15.97:0/3589627776 <== mon.2 v1:172.21.15.97:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 41758603 0 0) 0x7f496c004eb0 con 0x7f498813f710 2024-03-25T18:24:15.928 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:15.927+0000 7f497effd640 1 -- 172.21.15.97:0/3589627776 --> v1:172.21.15.97:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4988137010 con 0x7f498813f710 2024-03-25T18:24:15.928 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:15.927+0000 7f497effd640 1 -- 172.21.15.97:0/3589627776 <== mon.2 v1:172.21.15.97:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f4978001c00 con 0x7f498813f710 2024-03-25T18:24:15.928 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:15.927+0000 7f497effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:15.928 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:15.927+0000 7f497effd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:24:15.928 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:15.927+0000 7f497effd640 10 monclient(hunting): dump: 2024-03-25T18:24:15.928 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:24:15.928 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:24:15.928 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:24:15.928 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:24:15.928 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:15.928 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:24:15.928 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:24:15.928 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:24:15.929 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:24:15.929 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:24:15.929 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:15.927+0000 7f497effd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:24:15.937 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:15.936+0000 7f497effd640 1 -- 172.21.15.97:0/3589627776 <== mon.2 v1:172.21.15.97:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 708952314 0 0) 0x7f4988137010 con 0x7f498813f710 2024-03-25T18:24:15.937 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:15.936+0000 7f497effd640 1 -- 172.21.15.97:0/3589627776 >> v1:172.21.15.98:6789/0 conn(0x7f49880a98f0 legacy=0x7f4988130ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:15.937 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:15.936+0000 7f497effd640 1 -- 172.21.15.97:0/3589627776 >> v1:172.21.15.97:6789/0 conn(0x7f49881430c0 legacy=0x7f49881e2e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:15.937 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:15.936+0000 7f497effd640 10 monclient: _finish_hunting 0 2024-03-25T18:24:15.937 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:15.936+0000 7f497effd640 1 monclient: found mon.c 2024-03-25T18:24:15.937 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:15.936+0000 7f497effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:15.937 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:15.936+0000 7f497effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.97:6790/0 2024-03-25T18:24:15.937 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:15.936+0000 7f497effd640 1 -- 172.21.15.97:0/3589627776 --> v1:172.21.15.97:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4988135550 con 0x7f498813f710 2024-03-25T18:24:15.937 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:15.936+0000 7f497effd640 10 monclient: _finish_auth 0 2024-03-25T18:24:15.937 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:15.936+0000 7f497effd640 10 monclient: _check_auth_tickets 2024-03-25T18:24:15.937 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:15.936+0000 7f497effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:15.938 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:15.936+0000 7f4990789640 5 monclient: authenticate success, global_id 4577 2024-03-25T18:24:15.938 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:15.936+0000 7f4990789640 10 monclient: _renew_subs 2024-03-25T18:24:15.938 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:15.936+0000 7f4990789640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.97:6790/0 2024-03-25T18:24:15.938 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:15.936+0000 7f4990789640 1 -- 172.21.15.97:0/3589627776 --> v1:172.21.15.97:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4988135820 con 0x7f498813f710 2024-03-25T18:24:15.938 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:15.936+0000 7f4990789640 10 client.4577.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:24:15.938 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:15.936+0000 7f4990789640 10 monclient: _renew_subs 2024-03-25T18:24:15.938 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:15.936+0000 7f4990789640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.97:6790/0 2024-03-25T18:24:15.938 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:15.936+0000 7f4990789640 1 -- 172.21.15.97:0/3589627776 --> v1:172.21.15.97:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f498813b800 con 0x7f498813f710 2024-03-25T18:24:15.938 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:15.936+0000 7f4990789640 1 librados: init done 2024-03-25T18:24:15.939 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:15.937+0000 7f4990789640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:15.939 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:15.937+0000 7f4990789640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:15.939 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:15.937+0000 7f4990789640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.97:6790/0 2024-03-25T18:24:15.939 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:15.937+0000 7f4990789640 1 -- 172.21.15.97:0/3589627776 --> v1:172.21.15.97:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49880b7ac0 con 0x7f498813f710 2024-03-25T18:24:16.232 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:16.231+0000 7f497effd640 1 -- 172.21.15.97:0/3589627776 <== mon.2 v1:172.21.15.97:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4978004a90 con 0x7f498813f710 2024-03-25T18:24:16.232 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:16.231+0000 7f497effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:16.232 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:16.231+0000 7f497effd640 1 -- 172.21.15.97:0/3589627776 <== mon.2 v1:172.21.15.97:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f4978004e20 con 0x7f498813f710 2024-03-25T18:24:16.232 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:16.231+0000 7f497effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:16.232 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:16.231+0000 7f497effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:24:16.232 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:16.231+0000 7f497effd640 10 monclient: dump: 2024-03-25T18:24:16.233 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:24:16.233 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:24:16.233 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:24:16.233 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:24:16.233 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:16.233 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:24:16.233 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:24:16.233 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:24:16.233 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:24:16.233 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:24:16.233 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:16.231+0000 7f498cd22640 4 set_mon_vals no callback set 2024-03-25T18:24:16.234 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:16.232+0000 7f497effd640 1 -- 172.21.15.97:0/3589627776 <== mon.2 v1:172.21.15.97:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2377622137 0 0) 0x7f4978005040 con 0x7f498813f710 2024-03-25T18:24:16.234 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:16.233+0000 7f498e525640 10 monclient: get_auth_request con 0x7f496c06ddf0 auth_method 0 2024-03-25T18:24:16.234 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:16.233+0000 7f497effd640 10 client.4577.objecter ms_handle_connect 0x7f496c06ddf0 2024-03-25T18:24:16.239 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:16.239+0000 7f497effd640 1 -- 172.21.15.97:0/3589627776 <== mon.2 v1:172.21.15.97:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 430711918 0 0) 0x7f4978054930 con 0x7f498813f710 2024-03-25T18:24:16.240 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:16.239+0000 7f497effd640 10 client.4577.objecter ms_dispatch 0x7f4988000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:24:16.240 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:16.239+0000 7f497effd640 3 client.4577.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:24:16.240 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:16.239+0000 7f497effd640 3 client.4577.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:24:16.240 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:16.239+0000 7f497effd640 20 client.4577.objecter dump_active .. 0 homeless 2024-03-25T18:24:16.240 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:16.239+0000 7f497effd640 1 -- 172.21.15.97:0/3589627776 <== mon.2 v1:172.21.15.97:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4978050440 con 0x7f498813f710 2024-03-25T18:24:16.240 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:16.239+0000 7f497effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:16.240 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:16.239+0000 7f497effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:24:16.487 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:16.486+0000 7f4990789640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:16.487 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:16.486+0000 7f4990789640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:16.487 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:16.486+0000 7f4990789640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.97:6790/0 2024-03-25T18:24:16.487 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:16.486+0000 7f4990789640 1 -- 172.21.15.97:0/3589627776 --> v1:172.21.15.97:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4988136e50 con 0x7f498813f710 2024-03-25T18:24:16.594 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:16.593+0000 7f497effd640 1 -- 172.21.15.97:0/3589627776 <== mon.2 v1:172.21.15.97:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2169427412) 0x7f4978052370 con 0x7f498813f710 2024-03-25T18:24:16.594 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:16.593+0000 7f497effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:16.594 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:16.593+0000 7f497effd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:24:16.594 INFO:teuthology.misc.health.smithi097.stdout: 2024-03-25T18:24:16.594 INFO:teuthology.misc.health.smithi097.stdout:{"epoch":10,"fsid":"4c3d472c-79b8-486b-9e3d-2738d4306754","created":"2024-03-25T18:16:14.737309+0000","modified":"2024-03-25T18:16:20.357492+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:20.357492+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cb5152b9-262f-45c3-94b8-18d0e81df558","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6daf8793-0838-473d-a7d6-9a0d2894c75d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"136f1493-3b9c-4f43-a2e7-641482d69dae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"397d538c-4f64-4daf-9e8b-662d09f91327","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4f1df08-663f-4bda-93f8-3878ab7c24b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c4b18d70-a3ce-4b15-9b70-eae5525bd761","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8dd4033a-de0e-4f8a-96dc-59f3c1f825b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0077722e-ffa6-4a3d-9891-929d9869b286","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:24:16.597 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:16.596+0000 7f4990789640 10 librados: watch_flush enter 2024-03-25T18:24:16.597 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:16.596+0000 7f4990789640 10 librados: watch_flush exit 2024-03-25T18:24:16.598 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:16.596+0000 7f4990789640 20 client.4577.objecter shutdown clearing up homeless session... 2024-03-25T18:24:16.598 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:16.596+0000 7f4990789640 10 client.4577.objecter successfully canceled tick 2024-03-25T18:24:16.598 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:16.596+0000 7f4990789640 1 -- 172.21.15.97:0/3589627776 >> v1:172.21.15.98:6800/800387573 conn(0x7f496c06ddf0 legacy=0x7f496c0702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:16.598 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:16.596+0000 7f4990789640 10 monclient: shutdown 2024-03-25T18:24:16.598 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:16.596+0000 7f4990789640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:16.598 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:16.596+0000 7f4990789640 1 -- 172.21.15.97:0/3589627776 >> v1:172.21.15.97:6790/0 conn(0x7f498813f710 legacy=0x7f49881dab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:16.598 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:16.596+0000 7f4990789640 1 -- 172.21.15.97:0/3589627776 shutdown_connections 2024-03-25T18:24:16.598 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:16.596+0000 7f4990789640 1 -- 172.21.15.97:0/3589627776 >> 172.21.15.97:0/3589627776 conn(0x7f49880b7ea0 msgr2=0x7f4988074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:16.598 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:16.596+0000 7f4990789640 1 -- 172.21.15.97:0/3589627776 shutdown_connections 2024-03-25T18:24:16.598 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:16.596+0000 7f4990789640 1 -- 172.21.15.97:0/3589627776 wait complete. 2024-03-25T18:24:16.598 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:16.596+0000 7f4990789640 1 librados: shutdown 2024-03-25T18:24:16.609 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:24:23.415 DEBUG:teuthology.orchestra.run.smithi097:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:24:23.524 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.523+0000 7f490223b640 10 monclient: get_monmap_and_config 2024-03-25T18:24:23.524 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.523+0000 7f490223b640 10 monclient: build_initial_monmap 2024-03-25T18:24:23.524 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.523+0000 7f490223b640 10 monclient: monmap: 2024-03-25T18:24:23.524 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:24:23.524 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:24:23.524 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:24:23.523709+0000 2024-03-25T18:24:23.524 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:24:23.523709+0000 2024-03-25T18:24:23.524 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:23.524 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:24:23.524 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:24:23.524 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T18:24:23.524 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-b 2024-03-25T18:24:23.525 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:24:23.526 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.524+0000 7f490223b640 1 Processor -- start 2024-03-25T18:24:23.526 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.524+0000 7f490223b640 1 -- start start 2024-03-25T18:24:23.526 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.524+0000 7f490223b640 10 monclient: init 2024-03-25T18:24:23.526 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.525+0000 7f490223b640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:23.526 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.525+0000 7f490223b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:24:23.526 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.525+0000 7f490223b640 10 monclient(hunting): picked mon.noname-b con 0x7f48fc0a98f0 addr v1:172.21.15.97:6790/0 2024-03-25T18:24:23.526 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.525+0000 7f490223b640 10 monclient(hunting): picked mon.noname-c con 0x7f48fc13f710 addr v1:172.21.15.98:6789/0 2024-03-25T18:24:23.526 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.525+0000 7f490223b640 10 monclient(hunting): picked mon.noname-a con 0x7f48fc1430c0 addr v1:172.21.15.97:6789/0 2024-03-25T18:24:23.527 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.525+0000 7f490223b640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48fc142520 con 0x7f48fc1430c0 2024-03-25T18:24:23.527 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.525+0000 7f490223b640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48fc0b28a0 con 0x7f48fc13f710 2024-03-25T18:24:23.527 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.525+0000 7f490223b640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48fc0b2a30 con 0x7f48fc0a98f0 2024-03-25T18:24:23.527 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.525+0000 7f490223b640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:23.527 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.525+0000 7f490223b640 10 monclient(hunting): authenticate will time out at 8075.800101s 2024-03-25T18:24:23.535 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.535+0000 7f48fb7fe640 1 --1- >> v1:172.21.15.97:6790/0 conn(0x7f48fc0a98f0 0x7f48fc0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6790/0 says I am v1:172.21.15.97:59954/0 (socket says 172.21.15.97:59954) 2024-03-25T18:24:23.535 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.535+0000 7f48fb7fe640 1 -- 172.21.15.97:0/2446516069 learned_addr learned my addr 172.21.15.97:0/2446516069 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:24:23.549 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.549+0000 7f48fa7fc640 1 -- 172.21.15.97:0/2446516069 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 6788840 0 0) 0x7f48fc142520 con 0x7f48fc1430c0 2024-03-25T18:24:23.550 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.549+0000 7f48fa7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:24:23.550 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.549+0000 7f48fa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:23.550 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.549+0000 7f48fa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:23.550 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.549+0000 7f48fa7fc640 10 monclient(hunting): my global_id is 4584 2024-03-25T18:24:23.550 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.549+0000 7f48fa7fc640 1 -- 172.21.15.97:0/2446516069 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48e4003520 con 0x7f48fc1430c0 2024-03-25T18:24:23.560 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.559+0000 7f48fa7fc640 1 -- 172.21.15.97:0/2446516069 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3140513734 0 0) 0x7f48fc0b2a30 con 0x7f48fc0a98f0 2024-03-25T18:24:23.560 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.559+0000 7f48fa7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:24:23.560 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.559+0000 7f48fa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:23.560 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.559+0000 7f48fa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:23.560 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.559+0000 7f48fa7fc640 10 monclient(hunting): my global_id is 4580 2024-03-25T18:24:23.560 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.559+0000 7f48fa7fc640 1 -- 172.21.15.97:0/2446516069 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48fc142520 con 0x7f48fc0a98f0 2024-03-25T18:24:23.573 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.573+0000 7f48fa7fc640 1 -- 172.21.15.97:0/2446516069 <== mon.2 v1:172.21.15.97:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 816621841 0 0) 0x7f48fc142520 con 0x7f48fc0a98f0 2024-03-25T18:24:23.574 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.573+0000 7f48fa7fc640 1 -- 172.21.15.97:0/2446516069 >> v1:172.21.15.98:6789/0 conn(0x7f48fc13f710 legacy=0x7f48fc1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:23.574 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.573+0000 7f48fa7fc640 1 -- 172.21.15.97:0/2446516069 >> v1:172.21.15.97:6789/0 conn(0x7f48fc1430c0 legacy=0x7f48fc14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:23.574 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.573+0000 7f48fa7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:24:23.574 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.573+0000 7f48fa7fc640 1 monclient: found mon.noname-b 2024-03-25T18:24:23.574 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.573+0000 7f48fa7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:23.574 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.573+0000 7f48fa7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.97:6790/0 2024-03-25T18:24:23.574 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.573+0000 7f48fa7fc640 1 -- 172.21.15.97:0/2446516069 --> v1:172.21.15.97:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48fc0b44c0 con 0x7f48fc0a98f0 2024-03-25T18:24:23.574 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.573+0000 7f48fa7fc640 10 monclient: _finish_auth 0 2024-03-25T18:24:23.574 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.573+0000 7f48fa7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:24:23.574 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.573+0000 7f48fa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:23.574 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.573+0000 7f48fa7fc640 1 -- 172.21.15.97:0/2446516069 <== mon.2 v1:172.21.15.97:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f48ec001670 con 0x7f48fc0a98f0 2024-03-25T18:24:23.574 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.573+0000 7f48fa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:23.575 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.573+0000 7f48fa7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:24:23.575 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.573+0000 7f48fa7fc640 10 monclient: dump: 2024-03-25T18:24:23.575 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:24:23.575 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:24:23.575 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:24:23.575 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:24:23.575 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:23.575 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:24:23.575 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:24:23.575 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:24:23.575 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:24:23.575 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:24:23.575 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.573+0000 7f490223b640 5 monclient: authenticate success, global_id 4580 2024-03-25T18:24:23.575 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.573+0000 7f490223b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:24:23.595 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.594+0000 7f48fa7fc640 1 -- 172.21.15.97:0/2446516069 <== mon.2 v1:172.21.15.97:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f48ec0017d0 con 0x7f48fc0a98f0 2024-03-25T18:24:23.595 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.594+0000 7f48fa7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:23.595 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.595+0000 7f490223b640 10 monclient: get_monmap_and_config success 2024-03-25T18:24:23.595 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.594+0000 7f48fa7fc640 1 -- 172.21.15.97:0/2446516069 <== mon.2 v1:172.21.15.97:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f48ec001be0 con 0x7f48fc0a98f0 2024-03-25T18:24:23.596 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.595+0000 7f490223b640 4 set_mon_vals no callback set 2024-03-25T18:24:23.596 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.595+0000 7f48fa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:23.596 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.595+0000 7f48fa7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:24:23.596 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.595+0000 7f48fa7fc640 10 monclient: dump: 2024-03-25T18:24:23.596 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:24:23.596 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:24:23.596 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:24:23.596 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:24:23.596 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:23.596 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:24:23.596 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:24:23.596 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:24:23.597 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:24:23.597 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:24:23.599 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.595+0000 7f490223b640 10 monclient: shutdown 2024-03-25T18:24:23.599 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.595+0000 7f490223b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:23.599 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.595+0000 7f490223b640 1 -- 172.21.15.97:0/2446516069 >> v1:172.21.15.97:6790/0 conn(0x7f48fc0a98f0 legacy=0x7f48fc0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:23.599 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.595+0000 7f490223b640 1 -- 172.21.15.97:0/2446516069 shutdown_connections 2024-03-25T18:24:23.600 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.595+0000 7f490223b640 1 -- 172.21.15.97:0/2446516069 >> 172.21.15.97:0/2446516069 conn(0x7f48fc0b7ea0 msgr2=0x7f48fc0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:23.600 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.596+0000 7f490223b640 1 -- 172.21.15.97:0/2446516069 shutdown_connections 2024-03-25T18:24:23.600 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.596+0000 7f490223b640 1 -- 172.21.15.97:0/2446516069 wait complete. 2024-03-25T18:24:23.600 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.596+0000 7f490223b640 10 monclient: build_initial_monmap 2024-03-25T18:24:23.600 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.596+0000 7f490223b640 10 monclient: monmap: 2024-03-25T18:24:23.600 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:24:23.600 INFO:teuthology.misc.health.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:24:23.600 INFO:teuthology.misc.health.smithi097.stderr:last_changed 0.000000 2024-03-25T18:24:23.600 INFO:teuthology.misc.health.smithi097.stderr:created 0.000000 2024-03-25T18:24:23.600 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:23.600 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 1 2024-03-25T18:24:23.600 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:24:23.600 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T18:24:23.601 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-c 2024-03-25T18:24:23.601 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:24:23.601 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.597+0000 7f490223b640 1 librados: starting msgr at 2024-03-25T18:24:23.601 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.597+0000 7f490223b640 1 librados: starting objecter 2024-03-25T18:24:23.601 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.597+0000 7f490223b640 1 Processor -- start 2024-03-25T18:24:23.601 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.597+0000 7f490223b640 1 -- start start 2024-03-25T18:24:23.601 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.597+0000 7f490223b640 1 librados: setting wanted keys 2024-03-25T18:24:23.601 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.597+0000 7f490223b640 1 librados: calling monclient init 2024-03-25T18:24:23.601 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.597+0000 7f490223b640 10 monclient: init 2024-03-25T18:24:23.601 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.598+0000 7f490223b640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:23.601 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.598+0000 7f490223b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:24:23.601 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.598+0000 7f490223b640 10 monclient(hunting): picked mon.noname-b con 0x7f48fc0a98f0 addr v1:172.21.15.98:6789/0 2024-03-25T18:24:23.602 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.598+0000 7f490223b640 10 monclient(hunting): picked mon.noname-a con 0x7f48fc13f710 addr v1:172.21.15.97:6789/0 2024-03-25T18:24:23.602 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.598+0000 7f490223b640 10 monclient(hunting): picked mon.noname-c con 0x7f48fc1430c0 addr v1:172.21.15.97:6790/0 2024-03-25T18:24:23.602 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.598+0000 7f490223b640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48fc14e910 con 0x7f48fc13f710 2024-03-25T18:24:23.602 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.598+0000 7f490223b640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48fc0abb90 con 0x7f48fc0a98f0 2024-03-25T18:24:23.602 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.598+0000 7f490223b640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48fc14e5c0 con 0x7f48fc1430c0 2024-03-25T18:24:23.602 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.598+0000 7f490223b640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:23.602 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.598+0000 7f490223b640 10 monclient(hunting): authenticate will time out at 8075.873541s 2024-03-25T18:24:23.609 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.609+0000 7f48faffd640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f48fc13f710 0x7f48fc1d7870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:49084/0 (socket says 172.21.15.97:49084) 2024-03-25T18:24:23.609 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.609+0000 7f48faffd640 1 -- 172.21.15.97:0/130696925 learned_addr learned my addr 172.21.15.97:0/130696925 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:24:23.612 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.611+0000 7f48e3fff640 10 client.?.objecter ms_handle_connect 0x7f48fc0a98f0 2024-03-25T18:24:23.612 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.611+0000 7f48e3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:23.613 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.613+0000 7f48e3fff640 10 client.?.objecter ms_handle_connect 0x7f48fc13f710 2024-03-25T18:24:23.613 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.613+0000 7f48e3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:23.622 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.620+0000 7f48e3fff640 1 -- 172.21.15.97:0/130696925 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3025614015 0 0) 0x7f48fc14e910 con 0x7f48fc13f710 2024-03-25T18:24:23.622 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.620+0000 7f48e3fff640 10 monclient(hunting): _negotiate 2024-03-25T18:24:23.622 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.620+0000 7f48e3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:23.622 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.620+0000 7f48e3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:23.622 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.620+0000 7f48e3fff640 10 monclient(hunting): my global_id is 4587 2024-03-25T18:24:23.622 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.620+0000 7f48e3fff640 1 -- 172.21.15.97:0/130696925 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48e8004eb0 con 0x7f48fc13f710 2024-03-25T18:24:23.622 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.621+0000 7f48e3fff640 10 client.?.objecter ms_handle_connect 0x7f48fc1430c0 2024-03-25T18:24:23.622 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.621+0000 7f48e3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:23.624 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.624+0000 7f48e3fff640 1 -- 172.21.15.97:0/130696925 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 553017621 0 0) 0x7f48fc0abb90 con 0x7f48fc0a98f0 2024-03-25T18:24:23.624 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.624+0000 7f48e3fff640 10 monclient(hunting): _negotiate 2024-03-25T18:24:23.624 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.624+0000 7f48e3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:23.625 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.624+0000 7f48e3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:23.625 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.624+0000 7f48e3fff640 10 monclient(hunting): my global_id is 4594 2024-03-25T18:24:23.625 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.624+0000 7f48e3fff640 1 -- 172.21.15.97:0/130696925 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48fc14e910 con 0x7f48fc0a98f0 2024-03-25T18:24:23.636 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.635+0000 7f48e3fff640 1 -- 172.21.15.97:0/130696925 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2089984237 0 0) 0x7f48fc14e5c0 con 0x7f48fc1430c0 2024-03-25T18:24:23.636 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.635+0000 7f48e3fff640 10 monclient(hunting): _negotiate 2024-03-25T18:24:23.636 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.635+0000 7f48e3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:23.636 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.635+0000 7f48e3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:23.636 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.635+0000 7f48e3fff640 10 monclient(hunting): my global_id is 4583 2024-03-25T18:24:23.636 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.635+0000 7f48e3fff640 1 -- 172.21.15.97:0/130696925 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48fc0abb90 con 0x7f48fc1430c0 2024-03-25T18:24:23.639 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.638+0000 7f48e3fff640 1 -- 172.21.15.97:0/130696925 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2301514700 0 0) 0x7f48e8004eb0 con 0x7f48fc13f710 2024-03-25T18:24:23.639 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.638+0000 7f48e3fff640 1 -- 172.21.15.97:0/130696925 --> v1:172.21.15.97:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f48fc14e5c0 con 0x7f48fc13f710 2024-03-25T18:24:23.639 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.638+0000 7f48e3fff640 1 -- 172.21.15.97:0/130696925 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f48f0001a30 con 0x7f48fc13f710 2024-03-25T18:24:23.639 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.638+0000 7f48e3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:23.639 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.638+0000 7f48e3fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:24:23.639 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.638+0000 7f48e3fff640 10 monclient(hunting): dump: 2024-03-25T18:24:23.639 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:24:23.639 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:24:23.639 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:24:23.640 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:24:23.640 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:23.640 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:24:23.640 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:24:23.640 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:24:23.640 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:24:23.640 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:24:23.640 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.638+0000 7f48e3fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:24:23.641 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.641+0000 7f48e3fff640 1 -- 172.21.15.97:0/130696925 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1240881723 0 0) 0x7f48fc14e910 con 0x7f48fc0a98f0 2024-03-25T18:24:23.642 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.641+0000 7f48e3fff640 1 -- 172.21.15.97:0/130696925 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f48e8004eb0 con 0x7f48fc0a98f0 2024-03-25T18:24:23.642 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.641+0000 7f48e3fff640 1 -- 172.21.15.97:0/130696925 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f48ec001ae0 con 0x7f48fc0a98f0 2024-03-25T18:24:23.642 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.641+0000 7f48e3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:23.642 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.641+0000 7f48e3fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:24:23.642 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.641+0000 7f48e3fff640 10 monclient(hunting): dump: 2024-03-25T18:24:23.642 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:24:23.642 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:24:23.642 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:24:23.642 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:24:23.642 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:23.642 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:24:23.643 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:24:23.643 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:24:23.643 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:24:23.643 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:24:23.656 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.655+0000 7f48e3fff640 1 -- 172.21.15.97:0/130696925 <== mon.1 v1:172.21.15.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1691470389 0 0) 0x7f48e8004eb0 con 0x7f48fc0a98f0 2024-03-25T18:24:23.656 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.655+0000 7f48e3fff640 1 -- 172.21.15.97:0/130696925 >> v1:172.21.15.97:6790/0 conn(0x7f48fc1430c0 legacy=0x7f48fc1d7f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:23.656 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.655+0000 7f48e3fff640 1 -- 172.21.15.97:0/130696925 >> v1:172.21.15.97:6789/0 conn(0x7f48fc13f710 legacy=0x7f48fc1d7870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:23.656 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.655+0000 7f48e3fff640 10 monclient: _finish_hunting 0 2024-03-25T18:24:23.656 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.655+0000 7f48e3fff640 1 monclient: found mon.b 2024-03-25T18:24:23.657 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.655+0000 7f48e3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:23.657 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.655+0000 7f48e3fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:24:23.657 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.655+0000 7f48e3fff640 1 -- 172.21.15.97:0/130696925 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48fc138ad0 con 0x7f48fc0a98f0 2024-03-25T18:24:23.657 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.655+0000 7f48e3fff640 10 monclient: _finish_auth 0 2024-03-25T18:24:23.657 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.655+0000 7f48e3fff640 10 monclient: _check_auth_tickets 2024-03-25T18:24:23.657 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.655+0000 7f48e3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:23.657 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.655+0000 7f490223b640 5 monclient: authenticate success, global_id 4594 2024-03-25T18:24:23.657 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.655+0000 7f490223b640 10 monclient: _renew_subs 2024-03-25T18:24:23.657 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.655+0000 7f490223b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:24:23.657 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.655+0000 7f490223b640 1 -- 172.21.15.97:0/130696925 --> v1:172.21.15.98:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48fc138d70 con 0x7f48fc0a98f0 2024-03-25T18:24:23.657 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.655+0000 7f490223b640 10 client.4594.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:24:23.657 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.655+0000 7f490223b640 10 monclient: _renew_subs 2024-03-25T18:24:23.658 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.655+0000 7f490223b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:24:23.658 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.655+0000 7f490223b640 1 -- 172.21.15.97:0/130696925 --> v1:172.21.15.98:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f48fc1e32a0 con 0x7f48fc0a98f0 2024-03-25T18:24:23.658 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.655+0000 7f490223b640 1 librados: init done 2024-03-25T18:24:23.658 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.656+0000 7f490223b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:23.658 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.656+0000 7f490223b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:23.659 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.656+0000 7f490223b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:24:23.659 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.656+0000 7f490223b640 1 -- 172.21.15.97:0/130696925 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48fc0627b0 con 0x7f48fc0a98f0 2024-03-25T18:24:23.952 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.951+0000 7f48e3fff640 1 -- 172.21.15.97:0/130696925 <== mon.1 v1:172.21.15.98:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f48ec001210 con 0x7f48fc0a98f0 2024-03-25T18:24:23.952 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.951+0000 7f48e3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:23.952 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.951+0000 7f48e3fff640 1 -- 172.21.15.97:0/130696925 <== mon.1 v1:172.21.15.98:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f48ec004c30 con 0x7f48fc0a98f0 2024-03-25T18:24:23.952 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.951+0000 7f48e3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:23.952 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.951+0000 7f48e3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:24:23.952 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.951+0000 7f48e3fff640 10 monclient: dump: 2024-03-25T18:24:23.953 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:24:23.953 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:24:23.953 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:24:23.953 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:24:23.953 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:23.953 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:24:23.953 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:24:23.953 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:24:23.953 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:24:23.953 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:24:23.953 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.951+0000 7f48f9ffb640 4 set_mon_vals no callback set 2024-03-25T18:24:23.954 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.952+0000 7f48e3fff640 1 -- 172.21.15.97:0/130696925 <== mon.1 v1:172.21.15.98:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2377622137 0 0) 0x7f48ec01ac00 con 0x7f48fc0a98f0 2024-03-25T18:24:23.954 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.953+0000 7f48faffd640 10 monclient: get_auth_request con 0x7f48e806e040 auth_method 0 2024-03-25T18:24:23.954 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.953+0000 7f48e3fff640 10 client.4594.objecter ms_handle_connect 0x7f48e806e040 2024-03-25T18:24:23.959 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.958+0000 7f48e3fff640 1 -- 172.21.15.97:0/130696925 <== mon.1 v1:172.21.15.98:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 430711918 0 0) 0x7f48ec054880 con 0x7f48fc0a98f0 2024-03-25T18:24:23.959 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.958+0000 7f48e3fff640 10 client.4594.objecter ms_dispatch 0x7f48fc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:24:23.959 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.958+0000 7f48e3fff640 3 client.4594.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:24:23.959 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.958+0000 7f48e3fff640 3 client.4594.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:24:23.959 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.959+0000 7f48e3fff640 20 client.4594.objecter dump_active .. 0 homeless 2024-03-25T18:24:23.960 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.959+0000 7f48e3fff640 1 -- 172.21.15.97:0/130696925 <== mon.1 v1:172.21.15.98:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f48ec0501c0 con 0x7f48fc0a98f0 2024-03-25T18:24:23.960 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.959+0000 7f48e3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:23.960 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:23.959+0000 7f48e3fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:24:24.204 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:24.204+0000 7f490223b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:24.204 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:24.204+0000 7f490223b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:24.204 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:24.204+0000 7f490223b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:24:24.205 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:24.204+0000 7f490223b640 1 -- 172.21.15.97:0/130696925 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f48fc141c20 con 0x7f48fc0a98f0 2024-03-25T18:24:24.222 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:24.221+0000 7f48e3fff640 1 -- 172.21.15.97:0/130696925 <== mon.1 v1:172.21.15.98:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2169427412) 0x7f48ec0522c0 con 0x7f48fc0a98f0 2024-03-25T18:24:24.222 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:24.221+0000 7f48e3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:24.222 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:24.221+0000 7f48e3fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:24:24.222 INFO:teuthology.misc.health.smithi097.stdout: 2024-03-25T18:24:24.222 INFO:teuthology.misc.health.smithi097.stdout:{"epoch":10,"fsid":"4c3d472c-79b8-486b-9e3d-2738d4306754","created":"2024-03-25T18:16:14.737309+0000","modified":"2024-03-25T18:16:20.357492+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:20.357492+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cb5152b9-262f-45c3-94b8-18d0e81df558","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6daf8793-0838-473d-a7d6-9a0d2894c75d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"136f1493-3b9c-4f43-a2e7-641482d69dae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"397d538c-4f64-4daf-9e8b-662d09f91327","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4f1df08-663f-4bda-93f8-3878ab7c24b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c4b18d70-a3ce-4b15-9b70-eae5525bd761","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8dd4033a-de0e-4f8a-96dc-59f3c1f825b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0077722e-ffa6-4a3d-9891-929d9869b286","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:24:24.224 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:24.223+0000 7f490223b640 10 librados: watch_flush enter 2024-03-25T18:24:24.224 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:24.223+0000 7f490223b640 10 librados: watch_flush exit 2024-03-25T18:24:24.224 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:24.223+0000 7f490223b640 20 client.4594.objecter shutdown clearing up homeless session... 2024-03-25T18:24:24.225 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:24.223+0000 7f490223b640 10 client.4594.objecter successfully canceled tick 2024-03-25T18:24:24.225 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:24.223+0000 7f490223b640 1 -- 172.21.15.97:0/130696925 >> v1:172.21.15.98:6800/800387573 conn(0x7f48e806e040 legacy=0x7f48e8070500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:24.225 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:24.223+0000 7f490223b640 10 monclient: shutdown 2024-03-25T18:24:24.225 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:24.223+0000 7f490223b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:24.225 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:24.223+0000 7f490223b640 1 -- 172.21.15.97:0/130696925 >> v1:172.21.15.98:6789/0 conn(0x7f48fc0a98f0 legacy=0x7f48fc1383c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:24.225 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:24.223+0000 7f490223b640 1 -- 172.21.15.97:0/130696925 shutdown_connections 2024-03-25T18:24:24.225 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:24.223+0000 7f490223b640 1 -- 172.21.15.97:0/130696925 >> 172.21.15.97:0/130696925 conn(0x7f48fc0b7ea0 msgr2=0x7f48fc074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:24.225 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:24.224+0000 7f490223b640 1 -- 172.21.15.97:0/130696925 shutdown_connections 2024-03-25T18:24:24.225 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:24.224+0000 7f490223b640 1 -- 172.21.15.97:0/130696925 wait complete. 2024-03-25T18:24:24.225 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:24.224+0000 7f490223b640 1 librados: shutdown 2024-03-25T18:24:24.235 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:24:31.041 DEBUG:teuthology.orchestra.run.smithi097:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:24:31.152 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.150+0000 7efd00bbc640 10 monclient: get_monmap_and_config 2024-03-25T18:24:31.152 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.150+0000 7efd00bbc640 10 monclient: build_initial_monmap 2024-03-25T18:24:31.152 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.150+0000 7efd00bbc640 10 monclient: monmap: 2024-03-25T18:24:31.152 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:24:31.152 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:24:31.152 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:24:31.152033+0000 2024-03-25T18:24:31.152 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:24:31.152033+0000 2024-03-25T18:24:31.152 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:31.153 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:24:31.153 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:24:31.153 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T18:24:31.153 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-b 2024-03-25T18:24:31.153 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:24:31.153 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.151+0000 7efd00bbc640 1 Processor -- start 2024-03-25T18:24:31.153 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.151+0000 7efd00bbc640 1 -- start start 2024-03-25T18:24:31.153 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.151+0000 7efd00bbc640 10 monclient: init 2024-03-25T18:24:31.154 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.152+0000 7efd00bbc640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:31.154 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.152+0000 7efd00bbc640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:24:31.155 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.152+0000 7efd00bbc640 10 monclient(hunting): picked mon.noname-a con 0x7efcfc0a98f0 addr v1:172.21.15.97:6789/0 2024-03-25T18:24:31.155 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.152+0000 7efd00bbc640 10 monclient(hunting): picked mon.noname-c con 0x7efcfc13f710 addr v1:172.21.15.98:6789/0 2024-03-25T18:24:31.155 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.152+0000 7efd00bbc640 10 monclient(hunting): picked mon.noname-b con 0x7efcfc1430c0 addr v1:172.21.15.97:6790/0 2024-03-25T18:24:31.155 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.152+0000 7efd00bbc640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcfc142520 con 0x7efcfc0a98f0 2024-03-25T18:24:31.155 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.152+0000 7efd00bbc640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcfc0b28a0 con 0x7efcfc13f710 2024-03-25T18:24:31.155 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.152+0000 7efd00bbc640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcfc0b2a30 con 0x7efcfc1430c0 2024-03-25T18:24:31.155 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.152+0000 7efd00bbc640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:31.155 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.152+0000 7efd00bbc640 10 monclient(hunting): authenticate will time out at 8083.428728s 2024-03-25T18:24:31.157 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.155+0000 7efcf9d9b640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7efcfc13f710 0x7efcfc1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.97:41622/0 (socket says 172.21.15.97:41622) 2024-03-25T18:24:31.157 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.155+0000 7efcf9d9b640 1 -- 172.21.15.97:0/527008683 learned_addr learned my addr 172.21.15.97:0/527008683 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:24:31.164 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.162+0000 7efcf959a640 1 -- 172.21.15.97:0/527008683 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2770723179 0 0) 0x7efcfc0b28a0 con 0x7efcfc13f710 2024-03-25T18:24:31.164 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.162+0000 7efcf959a640 10 monclient(hunting): _negotiate 2024-03-25T18:24:31.164 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.162+0000 7efcf959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:31.164 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.162+0000 7efcf959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:31.164 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.162+0000 7efcf959a640 10 monclient(hunting): my global_id is 4597 2024-03-25T18:24:31.164 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.162+0000 7efcf959a640 1 -- 172.21.15.97:0/527008683 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efce4003520 con 0x7efcfc13f710 2024-03-25T18:24:31.170 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.168+0000 7efcf959a640 1 -- 172.21.15.97:0/527008683 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1353648932 0 0) 0x7efce4003520 con 0x7efcfc13f710 2024-03-25T18:24:31.170 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.168+0000 7efcf959a640 1 -- 172.21.15.97:0/527008683 >> v1:172.21.15.97:6790/0 conn(0x7efcfc1430c0 legacy=0x7efcfc14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:24:31.170 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.168+0000 7efcf959a640 1 -- 172.21.15.97:0/527008683 >> v1:172.21.15.97:6789/0 conn(0x7efcfc0a98f0 legacy=0x7efcfc0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:31.170 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.168+0000 7efcf959a640 10 monclient: _finish_hunting 0 2024-03-25T18:24:31.171 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.168+0000 7efcf959a640 1 monclient: found mon.noname-c 2024-03-25T18:24:31.171 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.168+0000 7efcf959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:31.171 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.168+0000 7efcf959a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.98:6789/0 2024-03-25T18:24:31.171 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.168+0000 7efcf959a640 1 -- 172.21.15.97:0/527008683 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcfc0b44c0 con 0x7efcfc13f710 2024-03-25T18:24:31.171 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.168+0000 7efcf959a640 10 monclient: _finish_auth 0 2024-03-25T18:24:31.171 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.168+0000 7efcf959a640 10 monclient: _check_auth_tickets 2024-03-25T18:24:31.171 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.168+0000 7efcf959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:31.171 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.168+0000 7efcf959a640 1 -- 172.21.15.97:0/527008683 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7efcf0001690 con 0x7efcfc13f710 2024-03-25T18:24:31.171 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.168+0000 7efcf959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:31.171 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.168+0000 7efcf959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:24:31.171 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.168+0000 7efcf959a640 10 monclient: dump: 2024-03-25T18:24:31.171 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:24:31.171 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:24:31.171 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:24:31.171 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:24:31.172 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:31.172 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:24:31.172 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:24:31.172 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:24:31.172 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:24:31.172 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:24:31.172 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.168+0000 7efd00bbc640 5 monclient: authenticate success, global_id 4597 2024-03-25T18:24:31.172 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.168+0000 7efd00bbc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:24:31.179 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.177+0000 7efcf959a640 1 -- 172.21.15.97:0/527008683 <== mon.1 v1:172.21.15.98:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efcf0001860 con 0x7efcfc13f710 2024-03-25T18:24:31.179 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.177+0000 7efcf959a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:31.179 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.177+0000 7efcf959a640 1 -- 172.21.15.97:0/527008683 <== mon.1 v1:172.21.15.98:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7efcf0001cb0 con 0x7efcfc13f710 2024-03-25T18:24:31.179 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.177+0000 7efcf959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:31.179 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.177+0000 7efcf959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:24:31.179 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.177+0000 7efcf959a640 10 monclient: dump: 2024-03-25T18:24:31.179 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:24:31.180 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:24:31.180 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:24:31.180 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:24:31.180 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:31.180 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:24:31.180 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:24:31.180 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:24:31.180 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:24:31.180 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:24:31.180 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.177+0000 7efd00bbc640 10 monclient: get_monmap_and_config success 2024-03-25T18:24:31.180 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.178+0000 7efd00bbc640 4 set_mon_vals no callback set 2024-03-25T18:24:31.181 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.178+0000 7efd00bbc640 10 monclient: shutdown 2024-03-25T18:24:31.181 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.178+0000 7efd00bbc640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:31.181 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.178+0000 7efd00bbc640 1 -- 172.21.15.97:0/527008683 >> v1:172.21.15.98:6789/0 conn(0x7efcfc13f710 legacy=0x7efcfc1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:31.181 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.179+0000 7efd00bbc640 1 -- 172.21.15.97:0/527008683 shutdown_connections 2024-03-25T18:24:31.181 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.179+0000 7efd00bbc640 1 -- 172.21.15.97:0/527008683 >> 172.21.15.97:0/527008683 conn(0x7efcfc0b7ea0 msgr2=0x7efcfc0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:31.181 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.179+0000 7efd00bbc640 1 -- 172.21.15.97:0/527008683 shutdown_connections 2024-03-25T18:24:31.181 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.179+0000 7efd00bbc640 1 -- 172.21.15.97:0/527008683 wait complete. 2024-03-25T18:24:31.183 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.180+0000 7efd00bbc640 10 monclient: build_initial_monmap 2024-03-25T18:24:31.183 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.180+0000 7efd00bbc640 10 monclient: monmap: 2024-03-25T18:24:31.183 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:24:31.183 INFO:teuthology.misc.health.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:24:31.183 INFO:teuthology.misc.health.smithi097.stderr:last_changed 0.000000 2024-03-25T18:24:31.183 INFO:teuthology.misc.health.smithi097.stderr:created 0.000000 2024-03-25T18:24:31.183 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:31.183 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 1 2024-03-25T18:24:31.183 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:24:31.183 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T18:24:31.184 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-c 2024-03-25T18:24:31.184 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:24:31.184 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.180+0000 7efd00bbc640 1 librados: starting msgr at 2024-03-25T18:24:31.184 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.180+0000 7efd00bbc640 1 librados: starting objecter 2024-03-25T18:24:31.184 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.180+0000 7efd00bbc640 1 Processor -- start 2024-03-25T18:24:31.184 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.180+0000 7efd00bbc640 1 -- start start 2024-03-25T18:24:31.184 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.180+0000 7efd00bbc640 1 librados: setting wanted keys 2024-03-25T18:24:31.184 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.180+0000 7efd00bbc640 1 librados: calling monclient init 2024-03-25T18:24:31.184 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.180+0000 7efd00bbc640 10 monclient: init 2024-03-25T18:24:31.184 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.181+0000 7efd00bbc640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:31.184 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.181+0000 7efd00bbc640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:24:31.184 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.181+0000 7efd00bbc640 10 monclient(hunting): picked mon.noname-c con 0x7efcfc0a98f0 addr v1:172.21.15.97:6790/0 2024-03-25T18:24:31.184 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.181+0000 7efd00bbc640 10 monclient(hunting): picked mon.noname-b con 0x7efcfc13f710 addr v1:172.21.15.98:6789/0 2024-03-25T18:24:31.184 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.181+0000 7efd00bbc640 10 monclient(hunting): picked mon.noname-a con 0x7efcfc1430c0 addr v1:172.21.15.97:6789/0 2024-03-25T18:24:31.185 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.181+0000 7efd00bbc640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcfc1359a0 con 0x7efcfc1430c0 2024-03-25T18:24:31.185 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.181+0000 7efd00bbc640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcfc1319b0 con 0x7efcfc13f710 2024-03-25T18:24:31.185 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.181+0000 7efd00bbc640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcfc0b7bb0 con 0x7efcfc0a98f0 2024-03-25T18:24:31.185 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.181+0000 7efd00bbc640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:31.185 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.181+0000 7efd00bbc640 10 monclient(hunting): authenticate will time out at 8083.457633s 2024-03-25T18:24:31.190 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.189+0000 7efcfad9d640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7efcfc1430c0 0x7efcfc1312a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:52124/0 (socket says 172.21.15.97:52124) 2024-03-25T18:24:31.190 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.189+0000 7efcfad9d640 1 -- 172.21.15.97:0/347338112 learned_addr learned my addr 172.21.15.97:0/347338112 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:24:31.191 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.189+0000 7efceaffd640 10 client.?.objecter ms_handle_connect 0x7efcfc1430c0 2024-03-25T18:24:31.191 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.189+0000 7efceaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:31.199 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.197+0000 7efceaffd640 10 client.?.objecter ms_handle_connect 0x7efcfc13f710 2024-03-25T18:24:31.199 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.197+0000 7efceaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:31.200 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.199+0000 7efceaffd640 10 client.?.objecter ms_handle_connect 0x7efcfc0a98f0 2024-03-25T18:24:31.201 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.199+0000 7efceaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:31.201 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.200+0000 7efceaffd640 1 -- 172.21.15.97:0/347338112 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3669193340 0 0) 0x7efcfc1359a0 con 0x7efcfc1430c0 2024-03-25T18:24:31.201 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.200+0000 7efceaffd640 10 monclient(hunting): _negotiate 2024-03-25T18:24:31.202 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.200+0000 7efceaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:31.202 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.200+0000 7efceaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:31.202 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.200+0000 7efceaffd640 10 monclient(hunting): my global_id is 4593 2024-03-25T18:24:31.202 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.200+0000 7efceaffd640 1 -- 172.21.15.97:0/347338112 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efcdc004eb0 con 0x7efcfc1430c0 2024-03-25T18:24:31.208 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.207+0000 7efceaffd640 1 -- 172.21.15.97:0/347338112 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3702381581 0 0) 0x7efcfc1319b0 con 0x7efcfc13f710 2024-03-25T18:24:31.208 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.207+0000 7efceaffd640 10 monclient(hunting): _negotiate 2024-03-25T18:24:31.208 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.207+0000 7efceaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:31.209 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.207+0000 7efceaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:31.209 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.207+0000 7efceaffd640 10 monclient(hunting): my global_id is 4600 2024-03-25T18:24:31.209 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.207+0000 7efceaffd640 1 -- 172.21.15.97:0/347338112 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efcfc1359a0 con 0x7efcfc13f710 2024-03-25T18:24:31.210 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.208+0000 7efceaffd640 1 -- 172.21.15.97:0/347338112 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1572846635 0 0) 0x7efcdc004eb0 con 0x7efcfc1430c0 2024-03-25T18:24:31.210 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.209+0000 7efceaffd640 1 -- 172.21.15.97:0/347338112 --> v1:172.21.15.97:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efcfc1319b0 con 0x7efcfc1430c0 2024-03-25T18:24:31.211 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.209+0000 7efceaffd640 1 -- 172.21.15.97:0/347338112 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7efcf4001ad0 con 0x7efcfc1430c0 2024-03-25T18:24:31.211 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.209+0000 7efceaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:31.211 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.209+0000 7efceaffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:24:31.211 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.209+0000 7efceaffd640 10 monclient(hunting): dump: 2024-03-25T18:24:31.211 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:24:31.211 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:24:31.211 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:24:31.211 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:24:31.211 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:31.211 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:24:31.211 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:24:31.211 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:24:31.211 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:24:31.211 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:24:31.211 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.209+0000 7efceaffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:24:31.212 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.210+0000 7efceaffd640 1 -- 172.21.15.97:0/347338112 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2722251874 0 0) 0x7efcfc0b7bb0 con 0x7efcfc0a98f0 2024-03-25T18:24:31.212 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.210+0000 7efceaffd640 10 monclient(hunting): _negotiate 2024-03-25T18:24:31.212 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.210+0000 7efceaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:31.212 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.210+0000 7efceaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:31.212 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.210+0000 7efceaffd640 10 monclient(hunting): my global_id is 4586 2024-03-25T18:24:31.212 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.210+0000 7efceaffd640 1 -- 172.21.15.97:0/347338112 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efcdc004eb0 con 0x7efcfc0a98f0 2024-03-25T18:24:31.220 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.218+0000 7efceaffd640 1 -- 172.21.15.97:0/347338112 <== mon.0 v1:172.21.15.97:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 271346749 0 0) 0x7efcfc1319b0 con 0x7efcfc1430c0 2024-03-25T18:24:31.220 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.218+0000 7efceaffd640 1 -- 172.21.15.97:0/347338112 >> v1:172.21.15.97:6790/0 conn(0x7efcfc0a98f0 legacy=0x7efcfc1d7600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:31.220 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.218+0000 7efceaffd640 1 -- 172.21.15.97:0/347338112 >> v1:172.21.15.98:6789/0 conn(0x7efcfc13f710 legacy=0x7efcfc130b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:31.220 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.218+0000 7efceaffd640 10 monclient: _finish_hunting 0 2024-03-25T18:24:31.220 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.218+0000 7efceaffd640 1 monclient: found mon.a 2024-03-25T18:24:31.220 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.218+0000 7efceaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:31.220 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.218+0000 7efceaffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:24:31.220 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.218+0000 7efceaffd640 1 -- 172.21.15.97:0/347338112 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcfc135070 con 0x7efcfc1430c0 2024-03-25T18:24:31.220 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.218+0000 7efceaffd640 10 monclient: _finish_auth 0 2024-03-25T18:24:31.220 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.218+0000 7efceaffd640 10 monclient: _check_auth_tickets 2024-03-25T18:24:31.220 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.218+0000 7efceaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:31.220 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.218+0000 7efd00bbc640 5 monclient: authenticate success, global_id 4593 2024-03-25T18:24:31.221 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.218+0000 7efd00bbc640 10 monclient: _renew_subs 2024-03-25T18:24:31.221 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.218+0000 7efd00bbc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:24:31.221 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.218+0000 7efd00bbc640 1 -- 172.21.15.97:0/347338112 --> v1:172.21.15.97:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcfc13b400 con 0x7efcfc1430c0 2024-03-25T18:24:31.221 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.218+0000 7efd00bbc640 10 client.4593.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:24:31.221 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.218+0000 7efd00bbc640 10 monclient: _renew_subs 2024-03-25T18:24:31.221 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.218+0000 7efd00bbc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:24:31.221 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.218+0000 7efd00bbc640 1 -- 172.21.15.97:0/347338112 --> v1:172.21.15.97:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efcfc0b75a0 con 0x7efcfc1430c0 2024-03-25T18:24:31.221 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.218+0000 7efd00bbc640 1 librados: init done 2024-03-25T18:24:31.222 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.219+0000 7efd00bbc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:31.222 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.219+0000 7efd00bbc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:31.222 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.219+0000 7efd00bbc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:24:31.222 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.219+0000 7efd00bbc640 1 -- 172.21.15.97:0/347338112 --> v1:172.21.15.97:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efcfc0627b0 con 0x7efcfc1430c0 2024-03-25T18:24:31.245 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.244+0000 7efceaffd640 1 -- 172.21.15.97:0/347338112 <== mon.0 v1:172.21.15.97:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efcf4001fe0 con 0x7efcfc1430c0 2024-03-25T18:24:31.245 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.244+0000 7efceaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:31.245 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.244+0000 7efceaffd640 1 -- 172.21.15.97:0/347338112 <== mon.0 v1:172.21.15.97:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7efcf4004d80 con 0x7efcfc1430c0 2024-03-25T18:24:31.246 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.244+0000 7efceaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:31.246 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.244+0000 7efceaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:24:31.246 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.244+0000 7efceaffd640 10 monclient: dump: 2024-03-25T18:24:31.246 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:24:31.246 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:24:31.246 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:24:31.246 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:24:31.246 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:31.246 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:24:31.246 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:24:31.246 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:24:31.246 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:24:31.246 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:24:31.246 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.244+0000 7efcf8d99640 4 set_mon_vals no callback set 2024-03-25T18:24:31.247 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.245+0000 7efceaffd640 1 -- 172.21.15.97:0/347338112 <== mon.0 v1:172.21.15.97:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2377622137 0 0) 0x7efcf401ad50 con 0x7efcfc1430c0 2024-03-25T18:24:31.247 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.245+0000 7efcfa59c640 10 monclient: get_auth_request con 0x7efcdc06ddb0 auth_method 0 2024-03-25T18:24:31.247 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.245+0000 7efceaffd640 10 client.4593.objecter ms_handle_connect 0x7efcdc06ddb0 2024-03-25T18:24:31.249 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.248+0000 7efceaffd640 1 -- 172.21.15.97:0/347338112 <== mon.0 v1:172.21.15.97:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 430711918 0 0) 0x7efcf4054a50 con 0x7efcfc1430c0 2024-03-25T18:24:31.250 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.248+0000 7efceaffd640 10 client.4593.objecter ms_dispatch 0x7efcfc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:24:31.250 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.248+0000 7efceaffd640 3 client.4593.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:24:31.250 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.248+0000 7efceaffd640 3 client.4593.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:24:31.250 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.248+0000 7efceaffd640 20 client.4593.objecter dump_active .. 0 homeless 2024-03-25T18:24:31.534 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.533+0000 7efceaffd640 1 -- 172.21.15.97:0/347338112 <== mon.0 v1:172.21.15.97:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7efcf4050390 con 0x7efcfc1430c0 2024-03-25T18:24:31.535 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.533+0000 7efceaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:31.535 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.533+0000 7efceaffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:24:31.756 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.754+0000 7efd00bbc640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:31.756 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.754+0000 7efd00bbc640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:31.756 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.754+0000 7efd00bbc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:24:31.756 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.754+0000 7efd00bbc640 1 -- 172.21.15.97:0/347338112 --> v1:172.21.15.97:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efcfc14f320 con 0x7efcfc1430c0 2024-03-25T18:24:31.789 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.787+0000 7efceaffd640 1 -- 172.21.15.97:0/347338112 <== mon.0 v1:172.21.15.97:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2169427412) 0x7efcf4052490 con 0x7efcfc1430c0 2024-03-25T18:24:31.789 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.787+0000 7efceaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:31.789 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.787+0000 7efceaffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:24:31.789 INFO:teuthology.misc.health.smithi097.stdout: 2024-03-25T18:24:31.789 INFO:teuthology.misc.health.smithi097.stdout:{"epoch":10,"fsid":"4c3d472c-79b8-486b-9e3d-2738d4306754","created":"2024-03-25T18:16:14.737309+0000","modified":"2024-03-25T18:16:20.357492+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:20.357492+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cb5152b9-262f-45c3-94b8-18d0e81df558","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6daf8793-0838-473d-a7d6-9a0d2894c75d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"136f1493-3b9c-4f43-a2e7-641482d69dae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"397d538c-4f64-4daf-9e8b-662d09f91327","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4f1df08-663f-4bda-93f8-3878ab7c24b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c4b18d70-a3ce-4b15-9b70-eae5525bd761","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8dd4033a-de0e-4f8a-96dc-59f3c1f825b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0077722e-ffa6-4a3d-9891-929d9869b286","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:24:31.792 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.790+0000 7efd00bbc640 10 librados: watch_flush enter 2024-03-25T18:24:31.792 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.790+0000 7efd00bbc640 10 librados: watch_flush exit 2024-03-25T18:24:31.792 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.790+0000 7efd00bbc640 20 client.4593.objecter shutdown clearing up homeless session... 2024-03-25T18:24:31.792 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.790+0000 7efd00bbc640 10 client.4593.objecter successfully canceled tick 2024-03-25T18:24:31.793 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.790+0000 7efd00bbc640 1 -- 172.21.15.97:0/347338112 >> v1:172.21.15.98:6800/800387573 conn(0x7efcdc06ddb0 legacy=0x7efcdc070270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:31.793 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.790+0000 7efd00bbc640 10 monclient: shutdown 2024-03-25T18:24:31.793 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.790+0000 7efd00bbc640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:31.793 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.790+0000 7efd00bbc640 1 -- 172.21.15.97:0/347338112 >> v1:172.21.15.97:6789/0 conn(0x7efcfc1430c0 legacy=0x7efcfc1312a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:31.793 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.790+0000 7efd00bbc640 1 -- 172.21.15.97:0/347338112 shutdown_connections 2024-03-25T18:24:31.793 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.790+0000 7efd00bbc640 1 -- 172.21.15.97:0/347338112 >> 172.21.15.97:0/347338112 conn(0x7efcfc0b7ea0 msgr2=0x7efcfc074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:31.793 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.790+0000 7efd00bbc640 1 -- 172.21.15.97:0/347338112 shutdown_connections 2024-03-25T18:24:31.793 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.790+0000 7efd00bbc640 1 -- 172.21.15.97:0/347338112 wait complete. 2024-03-25T18:24:31.793 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:31.790+0000 7efd00bbc640 1 librados: shutdown 2024-03-25T18:24:31.803 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:24:38.442 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:08:40.481 34931== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:38.452 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:08:40.491 34931== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:38.492 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:08:40.535 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:38.493 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:08:40.532 34931== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:38.502 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:08:40.546 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:38.512 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:08:40.551 34935== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:38.518 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:08:40.557 34935== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:38.526 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:08:40.569 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:38.545 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:08:40.583 34935== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:38.609 DEBUG:teuthology.orchestra.run.smithi097:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:24:38.716 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:38.714+0000 7fe198b3b640 10 monclient: get_monmap_and_config 2024-03-25T18:24:38.716 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:38.714+0000 7fe198b3b640 10 monclient: build_initial_monmap 2024-03-25T18:24:38.717 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:38.715+0000 7fe198b3b640 10 monclient: monmap: 2024-03-25T18:24:38.717 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:24:38.717 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:24:38.717 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:24:38.716265+0000 2024-03-25T18:24:38.717 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:24:38.716265+0000 2024-03-25T18:24:38.717 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:38.717 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:24:38.717 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:24:38.717 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T18:24:38.717 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-b 2024-03-25T18:24:38.717 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:24:38.718 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:38.715+0000 7fe198b3b640 1 Processor -- start 2024-03-25T18:24:38.719 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:38.716+0000 7fe198b3b640 1 -- start start 2024-03-25T18:24:38.719 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:38.716+0000 7fe198b3b640 10 monclient: init 2024-03-25T18:24:38.719 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:38.716+0000 7fe198b3b640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:38.719 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:38.716+0000 7fe198b3b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:24:38.719 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:38.716+0000 7fe198b3b640 10 monclient(hunting): picked mon.noname-c con 0x7fe1940a98f0 addr v1:172.21.15.98:6789/0 2024-03-25T18:24:38.719 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:38.716+0000 7fe198b3b640 10 monclient(hunting): picked mon.noname-a con 0x7fe19413f710 addr v1:172.21.15.97:6789/0 2024-03-25T18:24:38.719 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:38.716+0000 7fe198b3b640 10 monclient(hunting): picked mon.noname-b con 0x7fe1941430c0 addr v1:172.21.15.97:6790/0 2024-03-25T18:24:38.719 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:38.716+0000 7fe198b3b640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe194142520 con 0x7fe19413f710 2024-03-25T18:24:38.719 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:38.717+0000 7fe198b3b640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1940b28a0 con 0x7fe1940a98f0 2024-03-25T18:24:38.719 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:38.717+0000 7fe198b3b640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1940b2a30 con 0x7fe1941430c0 2024-03-25T18:24:38.720 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:38.717+0000 7fe198b3b640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:38.720 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:38.717+0000 7fe198b3b640 10 monclient(hunting): authenticate will time out at 8090.992908s 2024-03-25T18:24:38.723 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:38.721+0000 7fe19259c640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fe1940a98f0 0x7fe1940a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.97:46290/0 (socket says 172.21.15.97:46290) 2024-03-25T18:24:38.723 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:38.721+0000 7fe19259c640 1 -- 172.21.15.97:0/3977501372 learned_addr learned my addr 172.21.15.97:0/3977501372 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:24:38.734 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:38.732+0000 7fe191d9b640 1 -- 172.21.15.97:0/3977501372 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3640186576 0 0) 0x7fe1940b28a0 con 0x7fe1940a98f0 2024-03-25T18:24:38.734 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:38.732+0000 7fe191d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:24:38.734 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:38.732+0000 7fe191d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:38.734 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:38.732+0000 7fe191d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:38.734 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:38.732+0000 7fe191d9b640 10 monclient(hunting): my global_id is 4603 2024-03-25T18:24:38.734 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:38.732+0000 7fe191d9b640 1 -- 172.21.15.97:0/3977501372 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe174003520 con 0x7fe1940a98f0 2024-03-25T18:24:38.753 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:38.752+0000 7fe191d9b640 1 -- 172.21.15.97:0/3977501372 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3945099429 0 0) 0x7fe174003520 con 0x7fe1940a98f0 2024-03-25T18:24:38.754 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:38.752+0000 7fe191d9b640 1 -- 172.21.15.97:0/3977501372 >> v1:172.21.15.97:6790/0 conn(0x7fe1941430c0 legacy=0x7fe19414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:38.754 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:38.752+0000 7fe191d9b640 1 -- 172.21.15.97:0/3977501372 >> v1:172.21.15.97:6789/0 conn(0x7fe19413f710 legacy=0x7fe1941429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:38.754 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:38.752+0000 7fe191d9b640 10 monclient: _finish_hunting 0 2024-03-25T18:24:38.754 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:38.752+0000 7fe191d9b640 1 monclient: found mon.noname-c 2024-03-25T18:24:38.754 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:38.752+0000 7fe191d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:38.754 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:38.752+0000 7fe191d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.98:6789/0 2024-03-25T18:24:38.754 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:38.752+0000 7fe191d9b640 1 -- 172.21.15.97:0/3977501372 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1940b44c0 con 0x7fe1940a98f0 2024-03-25T18:24:38.754 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:38.752+0000 7fe191d9b640 10 monclient: _finish_auth 0 2024-03-25T18:24:38.754 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:38.752+0000 7fe191d9b640 10 monclient: _check_auth_tickets 2024-03-25T18:24:38.754 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:38.752+0000 7fe191d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:38.754 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:38.752+0000 7fe191d9b640 1 -- 172.21.15.97:0/3977501372 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7fe184001690 con 0x7fe1940a98f0 2024-03-25T18:24:38.754 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:38.752+0000 7fe191d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:38.754 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:38.752+0000 7fe191d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:24:38.755 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:38.752+0000 7fe191d9b640 10 monclient: dump: 2024-03-25T18:24:38.755 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:24:38.755 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:24:38.755 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:24:38.755 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:24:38.755 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:38.755 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:24:38.755 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:24:38.755 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:24:38.755 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:24:38.755 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:24:38.755 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:38.752+0000 7fe198b3b640 5 monclient: authenticate success, global_id 4603 2024-03-25T18:24:38.755 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:38.752+0000 7fe198b3b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:24:38.762 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:38.761+0000 7fe191d9b640 1 -- 172.21.15.97:0/3977501372 <== mon.1 v1:172.21.15.98:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe1840017f0 con 0x7fe1940a98f0 2024-03-25T18:24:38.763 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:38.761+0000 7fe191d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:38.763 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:38.761+0000 7fe191d9b640 1 -- 172.21.15.97:0/3977501372 <== mon.1 v1:172.21.15.98:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7fe184001c00 con 0x7fe1940a98f0 2024-03-25T18:24:38.763 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:38.761+0000 7fe198b3b640 10 monclient: get_monmap_and_config success 2024-03-25T18:24:38.763 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:38.761+0000 7fe198b3b640 4 set_mon_vals no callback set 2024-03-25T18:24:38.763 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:38.761+0000 7fe191d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:38.763 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:38.761+0000 7fe191d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:24:38.763 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:38.761+0000 7fe191d9b640 10 monclient: dump: 2024-03-25T18:24:38.763 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:24:38.763 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:24:38.763 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:24:38.763 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:24:38.763 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:38.763 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:24:38.763 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:24:38.764 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:24:38.764 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:24:38.764 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:24:38.766 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:38.761+0000 7fe198b3b640 10 monclient: shutdown 2024-03-25T18:24:38.766 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:38.762+0000 7fe198b3b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:38.766 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:38.762+0000 7fe198b3b640 1 -- 172.21.15.97:0/3977501372 >> v1:172.21.15.98:6789/0 conn(0x7fe1940a98f0 legacy=0x7fe1940a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:38.766 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:38.762+0000 7fe198b3b640 1 -- 172.21.15.97:0/3977501372 shutdown_connections 2024-03-25T18:24:38.766 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:38.762+0000 7fe198b3b640 1 -- 172.21.15.97:0/3977501372 >> 172.21.15.97:0/3977501372 conn(0x7fe1940b7ea0 msgr2=0x7fe1940b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:38.766 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:38.762+0000 7fe198b3b640 1 -- 172.21.15.97:0/3977501372 shutdown_connections 2024-03-25T18:24:38.766 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:38.762+0000 7fe198b3b640 1 -- 172.21.15.97:0/3977501372 wait complete. 2024-03-25T18:24:38.766 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:38.763+0000 7fe198b3b640 10 monclient: build_initial_monmap 2024-03-25T18:24:38.766 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:38.763+0000 7fe198b3b640 10 monclient: monmap: 2024-03-25T18:24:38.766 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:24:38.766 INFO:teuthology.misc.health.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:24:38.767 INFO:teuthology.misc.health.smithi097.stderr:last_changed 0.000000 2024-03-25T18:24:38.767 INFO:teuthology.misc.health.smithi097.stderr:created 0.000000 2024-03-25T18:24:38.767 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:38.767 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 1 2024-03-25T18:24:38.767 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:24:38.767 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T18:24:38.767 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-c 2024-03-25T18:24:38.767 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:24:38.767 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:38.763+0000 7fe198b3b640 1 librados: starting msgr at 2024-03-25T18:24:38.767 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:38.763+0000 7fe198b3b640 1 librados: starting objecter 2024-03-25T18:24:38.767 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:38.763+0000 7fe198b3b640 1 Processor -- start 2024-03-25T18:24:38.767 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:38.763+0000 7fe198b3b640 1 -- start start 2024-03-25T18:24:38.767 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:38.763+0000 7fe198b3b640 1 librados: setting wanted keys 2024-03-25T18:24:38.767 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:38.763+0000 7fe198b3b640 1 librados: calling monclient init 2024-03-25T18:24:38.767 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:38.763+0000 7fe198b3b640 10 monclient: init 2024-03-25T18:24:38.768 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:38.764+0000 7fe198b3b640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:38.768 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:38.764+0000 7fe198b3b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:24:38.768 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:38.764+0000 7fe198b3b640 10 monclient(hunting): picked mon.noname-a con 0x7fe1940a98f0 addr v1:172.21.15.97:6789/0 2024-03-25T18:24:38.768 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:38.764+0000 7fe198b3b640 10 monclient(hunting): picked mon.noname-b con 0x7fe19413f710 addr v1:172.21.15.98:6789/0 2024-03-25T18:24:38.768 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:38.764+0000 7fe198b3b640 10 monclient(hunting): picked mon.noname-c con 0x7fe1941430c0 addr v1:172.21.15.97:6790/0 2024-03-25T18:24:38.768 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:38.764+0000 7fe198b3b640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe194138b10 con 0x7fe1940a98f0 2024-03-25T18:24:38.768 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:38.764+0000 7fe198b3b640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe194136e10 con 0x7fe19413f710 2024-03-25T18:24:38.768 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:38.764+0000 7fe198b3b640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe194138d60 con 0x7fe1941430c0 2024-03-25T18:24:38.768 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:38.764+0000 7fe198b3b640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:38.768 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:38.764+0000 7fe198b3b640 10 monclient(hunting): authenticate will time out at 8091.040216s 2024-03-25T18:24:38.777 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:38.775+0000 7fe19259c640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fe1940a98f0 0x7fe194137560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:56764/0 (socket says 172.21.15.97:56764) 2024-03-25T18:24:38.777 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:38.775+0000 7fe19259c640 1 -- 172.21.15.97:0/3123366246 learned_addr learned my addr 172.21.15.97:0/3123366246 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:24:38.781 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:38.779+0000 7fe18affd640 10 client.?.objecter ms_handle_connect 0x7fe1940a98f0 2024-03-25T18:24:38.781 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:38.779+0000 7fe18affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:38.788 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:38.787+0000 7fe18affd640 10 client.?.objecter ms_handle_connect 0x7fe19413f710 2024-03-25T18:24:38.788 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:38.787+0000 7fe18affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:38.793 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:38.792+0000 7fe18affd640 1 -- 172.21.15.97:0/3123366246 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1852929624 0 0) 0x7fe194136e10 con 0x7fe19413f710 2024-03-25T18:24:38.793 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:38.792+0000 7fe18affd640 10 monclient(hunting): _negotiate 2024-03-25T18:24:38.794 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:38.792+0000 7fe18affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:38.794 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:38.792+0000 7fe18affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:38.794 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:38.792+0000 7fe18affd640 10 monclient(hunting): my global_id is 4606 2024-03-25T18:24:38.794 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:38.792+0000 7fe18affd640 1 -- 172.21.15.97:0/3123366246 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe178004eb0 con 0x7fe19413f710 2024-03-25T18:24:38.802 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:38.800+0000 7fe18affd640 1 -- 172.21.15.97:0/3123366246 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2918350333 0 0) 0x7fe194138b10 con 0x7fe1940a98f0 2024-03-25T18:24:38.802 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:38.800+0000 7fe18affd640 10 monclient(hunting): _negotiate 2024-03-25T18:24:38.802 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:38.800+0000 7fe18affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:38.802 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:38.800+0000 7fe18affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:38.802 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:38.800+0000 7fe18affd640 10 monclient(hunting): my global_id is 4599 2024-03-25T18:24:38.802 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:38.800+0000 7fe18affd640 1 -- 172.21.15.97:0/3123366246 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe194136e10 con 0x7fe1940a98f0 2024-03-25T18:24:38.804 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:38.802+0000 7fe18affd640 1 -- 172.21.15.97:0/3123366246 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4000966558 0 0) 0x7fe178004eb0 con 0x7fe19413f710 2024-03-25T18:24:38.804 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:38.802+0000 7fe18affd640 1 -- 172.21.15.97:0/3123366246 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe194138b10 con 0x7fe19413f710 2024-03-25T18:24:38.804 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:38.802+0000 7fe18affd640 1 -- 172.21.15.97:0/3123366246 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7fe17c001930 con 0x7fe19413f710 2024-03-25T18:24:38.804 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:38.802+0000 7fe18affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:38.804 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:38.802+0000 7fe18affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:24:38.804 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:38.802+0000 7fe18affd640 10 monclient(hunting): dump: 2024-03-25T18:24:38.804 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:24:38.805 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:24:38.805 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:24:38.805 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:24:38.805 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:38.805 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:24:38.805 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:24:38.805 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:24:38.805 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:24:38.805 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:24:38.805 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:38.803+0000 7fe18affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:24:38.815 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:38.813+0000 7fe18affd640 10 client.?.objecter ms_handle_connect 0x7fe1941430c0 2024-03-25T18:24:38.815 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:38.813+0000 7fe18affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:38.817 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:38.814+0000 7fe18affd640 1 -- 172.21.15.97:0/3123366246 <== mon.1 v1:172.21.15.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3477418790 0 0) 0x7fe194138b10 con 0x7fe19413f710 2024-03-25T18:24:38.817 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:38.814+0000 7fe18affd640 1 -- 172.21.15.97:0/3123366246 >> v1:172.21.15.97:6790/0 conn(0x7fe1941430c0 legacy=0x7fe1941e29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:38.817 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:38.814+0000 7fe18affd640 1 -- 172.21.15.97:0/3123366246 >> v1:172.21.15.97:6789/0 conn(0x7fe1940a98f0 legacy=0x7fe194137560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:38.817 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:38.814+0000 7fe18affd640 10 monclient: _finish_hunting 0 2024-03-25T18:24:38.817 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:38.814+0000 7fe18affd640 1 monclient: found mon.b 2024-03-25T18:24:38.817 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:38.815+0000 7fe18affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:38.817 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:38.815+0000 7fe18affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:24:38.817 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:38.815+0000 7fe18affd640 1 -- 172.21.15.97:0/3123366246 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe194139040 con 0x7fe19413f710 2024-03-25T18:24:38.817 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:38.815+0000 7fe18affd640 10 monclient: _finish_auth 0 2024-03-25T18:24:38.817 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:38.815+0000 7fe18affd640 10 monclient: _check_auth_tickets 2024-03-25T18:24:38.817 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:38.815+0000 7fe18affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:38.817 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:38.815+0000 7fe198b3b640 5 monclient: authenticate success, global_id 4606 2024-03-25T18:24:38.817 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:38.815+0000 7fe198b3b640 10 monclient: _renew_subs 2024-03-25T18:24:38.817 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:38.815+0000 7fe198b3b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:24:38.818 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:38.815+0000 7fe198b3b640 1 -- 172.21.15.97:0/3123366246 --> v1:172.21.15.98:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe194139380 con 0x7fe19413f710 2024-03-25T18:24:38.818 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:38.815+0000 7fe198b3b640 10 client.4606.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:24:38.818 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:38.815+0000 7fe198b3b640 10 monclient: _renew_subs 2024-03-25T18:24:38.818 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:38.815+0000 7fe198b3b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:24:38.818 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:38.815+0000 7fe198b3b640 1 -- 172.21.15.97:0/3123366246 --> v1:172.21.15.98:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe194139920 con 0x7fe19413f710 2024-03-25T18:24:38.818 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:38.815+0000 7fe198b3b640 1 librados: init done 2024-03-25T18:24:38.818 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:38.816+0000 7fe198b3b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:38.819 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:38.816+0000 7fe198b3b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:38.819 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:38.816+0000 7fe198b3b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:24:38.819 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:38.816+0000 7fe198b3b640 1 -- 172.21.15.97:0/3123366246 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1940627b0 con 0x7fe19413f710 2024-03-25T18:24:39.103 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:39.102+0000 7fe18affd640 1 -- 172.21.15.97:0/3123366246 <== mon.1 v1:172.21.15.98:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe17c003610 con 0x7fe19413f710 2024-03-25T18:24:39.104 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:39.102+0000 7fe18affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:39.104 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:39.102+0000 7fe18affd640 1 -- 172.21.15.97:0/3123366246 <== mon.1 v1:172.21.15.98:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7fe17c003a20 con 0x7fe19413f710 2024-03-25T18:24:39.104 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:39.102+0000 7fe18affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:39.104 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:39.102+0000 7fe18affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:24:39.104 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:39.102+0000 7fe18affd640 10 monclient: dump: 2024-03-25T18:24:39.104 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:24:39.104 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:24:39.104 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:24:39.104 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:24:39.104 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:39.104 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:24:39.104 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:24:39.104 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:24:39.104 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:24:39.104 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:24:39.105 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:39.102+0000 7fe19159a640 4 set_mon_vals no callback set 2024-03-25T18:24:39.105 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:39.103+0000 7fe18affd640 1 -- 172.21.15.97:0/3123366246 <== mon.1 v1:172.21.15.98:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2377622137 0 0) 0x7fe17c003c40 con 0x7fe19413f710 2024-03-25T18:24:39.105 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:39.103+0000 7fe19259c640 10 monclient: get_auth_request con 0x7fe17806dd10 auth_method 0 2024-03-25T18:24:39.105 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:39.104+0000 7fe18affd640 10 client.4606.objecter ms_handle_connect 0x7fe17806dd10 2024-03-25T18:24:39.112 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:39.110+0000 7fe18affd640 1 -- 172.21.15.97:0/3123366246 <== mon.1 v1:172.21.15.98:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 430711918 0 0) 0x7fe17c053610 con 0x7fe19413f710 2024-03-25T18:24:39.112 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:39.110+0000 7fe18affd640 10 client.4606.objecter ms_dispatch 0x7fe194000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:24:39.112 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:39.110+0000 7fe18affd640 3 client.4606.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:24:39.112 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:39.110+0000 7fe18affd640 3 client.4606.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:24:39.112 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:39.110+0000 7fe18affd640 20 client.4606.objecter dump_active .. 0 homeless 2024-03-25T18:24:39.112 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:39.110+0000 7fe18affd640 1 -- 172.21.15.97:0/3123366246 <== mon.1 v1:172.21.15.98:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe17c04f120 con 0x7fe19413f710 2024-03-25T18:24:39.112 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:39.110+0000 7fe18affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:39.112 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:39.110+0000 7fe18affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:24:39.360 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:39.359+0000 7fe198b3b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:39.360 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:39.359+0000 7fe198b3b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:39.360 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:39.359+0000 7fe198b3b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:24:39.360 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:39.359+0000 7fe198b3b640 1 -- 172.21.15.97:0/3123366246 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe1941391d0 con 0x7fe19413f710 2024-03-25T18:24:39.391 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:39.390+0000 7fe18affd640 1 -- 172.21.15.97:0/3123366246 <== mon.1 v1:172.21.15.98:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2169427412) 0x7fe17c051050 con 0x7fe19413f710 2024-03-25T18:24:39.391 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:39.390+0000 7fe18affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:39.391 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:39.390+0000 7fe18affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:24:39.391 INFO:teuthology.misc.health.smithi097.stdout: 2024-03-25T18:24:39.391 INFO:teuthology.misc.health.smithi097.stdout:{"epoch":10,"fsid":"4c3d472c-79b8-486b-9e3d-2738d4306754","created":"2024-03-25T18:16:14.737309+0000","modified":"2024-03-25T18:16:20.357492+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:20.357492+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cb5152b9-262f-45c3-94b8-18d0e81df558","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6daf8793-0838-473d-a7d6-9a0d2894c75d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"136f1493-3b9c-4f43-a2e7-641482d69dae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"397d538c-4f64-4daf-9e8b-662d09f91327","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4f1df08-663f-4bda-93f8-3878ab7c24b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c4b18d70-a3ce-4b15-9b70-eae5525bd761","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8dd4033a-de0e-4f8a-96dc-59f3c1f825b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0077722e-ffa6-4a3d-9891-929d9869b286","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:24:39.394 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:39.392+0000 7fe198b3b640 10 librados: watch_flush enter 2024-03-25T18:24:39.394 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:39.392+0000 7fe198b3b640 10 librados: watch_flush exit 2024-03-25T18:24:39.394 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:39.392+0000 7fe198b3b640 20 client.4606.objecter shutdown clearing up homeless session... 2024-03-25T18:24:39.395 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:39.392+0000 7fe198b3b640 10 client.4606.objecter successfully canceled tick 2024-03-25T18:24:39.395 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:39.392+0000 7fe198b3b640 1 -- 172.21.15.97:0/3123366246 >> v1:172.21.15.98:6800/800387573 conn(0x7fe17806dd10 legacy=0x7fe1780701d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:39.395 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:39.392+0000 7fe198b3b640 10 monclient: shutdown 2024-03-25T18:24:39.395 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:39.392+0000 7fe198b3b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:39.395 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:39.392+0000 7fe198b3b640 1 -- 172.21.15.97:0/3123366246 >> v1:172.21.15.98:6789/0 conn(0x7fe19413f710 legacy=0x7fe1941df270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:39.395 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:39.392+0000 7fe198b3b640 1 -- 172.21.15.97:0/3123366246 shutdown_connections 2024-03-25T18:24:39.395 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:39.392+0000 7fe198b3b640 1 -- 172.21.15.97:0/3123366246 >> 172.21.15.97:0/3123366246 conn(0x7fe1940b7ea0 msgr2=0x7fe194074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:39.395 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:39.392+0000 7fe198b3b640 1 -- 172.21.15.97:0/3123366246 shutdown_connections 2024-03-25T18:24:39.395 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:39.393+0000 7fe198b3b640 1 -- 172.21.15.97:0/3123366246 wait complete. 2024-03-25T18:24:39.395 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:39.393+0000 7fe198b3b640 1 librados: shutdown 2024-03-25T18:24:39.405 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:24:46.212 DEBUG:teuthology.orchestra.run.smithi097:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:24:46.332 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.330+0000 7fd24dc9d640 10 monclient: get_monmap_and_config 2024-03-25T18:24:46.332 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.330+0000 7fd24dc9d640 10 monclient: build_initial_monmap 2024-03-25T18:24:46.332 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.330+0000 7fd24dc9d640 10 monclient: monmap: 2024-03-25T18:24:46.332 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:24:46.332 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:24:46.332 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:24:46.331483+0000 2024-03-25T18:24:46.332 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:24:46.331483+0000 2024-03-25T18:24:46.332 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:46.332 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:24:46.332 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:24:46.332 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T18:24:46.333 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-b 2024-03-25T18:24:46.333 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:24:46.334 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.331+0000 7fd24dc9d640 1 Processor -- start 2024-03-25T18:24:46.334 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.331+0000 7fd24dc9d640 1 -- start start 2024-03-25T18:24:46.334 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.331+0000 7fd24dc9d640 10 monclient: init 2024-03-25T18:24:46.334 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.332+0000 7fd24dc9d640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:46.335 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.332+0000 7fd24dc9d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:24:46.335 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.332+0000 7fd24dc9d640 10 monclient(hunting): picked mon.noname-b con 0x7fd24806ea40 addr v1:172.21.15.97:6790/0 2024-03-25T18:24:46.335 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.332+0000 7fd24dc9d640 10 monclient(hunting): picked mon.noname-c con 0x7fd248071800 addr v1:172.21.15.98:6789/0 2024-03-25T18:24:46.335 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.332+0000 7fd24dc9d640 10 monclient(hunting): picked mon.noname-a con 0x7fd248071c40 addr v1:172.21.15.97:6789/0 2024-03-25T18:24:46.335 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.332+0000 7fd24dc9d640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd24806f1f0 con 0x7fd248071c40 2024-03-25T18:24:46.335 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.332+0000 7fd24dc9d640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd248070d00 con 0x7fd248071800 2024-03-25T18:24:46.335 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.332+0000 7fd24dc9d640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2480720b0 con 0x7fd24806ea40 2024-03-25T18:24:46.335 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.332+0000 7fd24dc9d640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:46.335 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.332+0000 7fd24dc9d640 10 monclient(hunting): authenticate will time out at 8098.608863s 2024-03-25T18:24:46.344 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.343+0000 7fd247fff640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fd248071c40 0x7fd24814a7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:39856/0 (socket says 172.21.15.97:39856) 2024-03-25T18:24:46.345 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.343+0000 7fd247fff640 1 -- 172.21.15.97:0/3375821700 learned_addr learned my addr 172.21.15.97:0/3375821700 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:24:46.356 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.355+0000 7fd2467fc640 1 -- 172.21.15.97:0/3375821700 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 251975664 0 0) 0x7fd24806f1f0 con 0x7fd248071c40 2024-03-25T18:24:46.356 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.355+0000 7fd2467fc640 10 monclient(hunting): _negotiate 2024-03-25T18:24:46.356 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.355+0000 7fd2467fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:46.356 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.355+0000 7fd2467fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:46.356 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.355+0000 7fd2467fc640 10 monclient(hunting): my global_id is 4602 2024-03-25T18:24:46.357 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.355+0000 7fd2467fc640 1 -- 172.21.15.97:0/3375821700 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd230003440 con 0x7fd248071c40 2024-03-25T18:24:46.357 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.355+0000 7fd2467fc640 1 -- 172.21.15.97:0/3375821700 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1092078047 0 0) 0x7fd2480720b0 con 0x7fd24806ea40 2024-03-25T18:24:46.357 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.355+0000 7fd2467fc640 10 monclient(hunting): _negotiate 2024-03-25T18:24:46.357 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.355+0000 7fd2467fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:46.357 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.355+0000 7fd2467fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:46.357 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.355+0000 7fd2467fc640 10 monclient(hunting): my global_id is 4592 2024-03-25T18:24:46.357 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.355+0000 7fd2467fc640 1 -- 172.21.15.97:0/3375821700 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd24806f1f0 con 0x7fd24806ea40 2024-03-25T18:24:46.359 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.357+0000 7fd2467fc640 1 -- 172.21.15.97:0/3375821700 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 98503170 0 0) 0x7fd248070d00 con 0x7fd248071800 2024-03-25T18:24:46.359 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.357+0000 7fd2467fc640 10 monclient(hunting): _negotiate 2024-03-25T18:24:46.359 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.357+0000 7fd2467fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:46.359 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.357+0000 7fd2467fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:46.359 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.357+0000 7fd2467fc640 10 monclient(hunting): my global_id is 4609 2024-03-25T18:24:46.359 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.357+0000 7fd2467fc640 1 -- 172.21.15.97:0/3375821700 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2480720b0 con 0x7fd248071800 2024-03-25T18:24:46.361 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.359+0000 7fd2467fc640 1 -- 172.21.15.97:0/3375821700 <== mon.2 v1:172.21.15.97:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2361695747 0 0) 0x7fd24806f1f0 con 0x7fd24806ea40 2024-03-25T18:24:46.361 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.359+0000 7fd2467fc640 1 -- 172.21.15.97:0/3375821700 >> v1:172.21.15.98:6789/0 conn(0x7fd248071800 legacy=0x7fd248147070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:46.361 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.359+0000 7fd2467fc640 1 -- 172.21.15.97:0/3375821700 >> v1:172.21.15.97:6789/0 conn(0x7fd248071c40 legacy=0x7fd24814a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:46.361 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.359+0000 7fd2467fc640 10 monclient: _finish_hunting 0 2024-03-25T18:24:46.362 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.359+0000 7fd2467fc640 1 monclient: found mon.noname-b 2024-03-25T18:24:46.362 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.359+0000 7fd2467fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:46.362 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.359+0000 7fd2467fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.97:6790/0 2024-03-25T18:24:46.362 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.359+0000 7fd2467fc640 1 -- 172.21.15.97:0/3375821700 --> v1:172.21.15.97:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd248072780 con 0x7fd24806ea40 2024-03-25T18:24:46.362 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.359+0000 7fd2467fc640 10 monclient: _finish_auth 0 2024-03-25T18:24:46.362 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.359+0000 7fd2467fc640 10 monclient: _check_auth_tickets 2024-03-25T18:24:46.362 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.359+0000 7fd2467fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:46.362 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.359+0000 7fd2467fc640 1 -- 172.21.15.97:0/3375821700 <== mon.2 v1:172.21.15.97:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7fd23c001670 con 0x7fd24806ea40 2024-03-25T18:24:46.362 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.359+0000 7fd2467fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:46.362 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.359+0000 7fd2467fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:24:46.362 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.359+0000 7fd2467fc640 10 monclient: dump: 2024-03-25T18:24:46.362 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:24:46.362 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:24:46.362 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:24:46.363 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:24:46.363 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:46.363 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:24:46.363 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:24:46.363 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:24:46.363 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:24:46.363 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:24:46.363 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.359+0000 7fd24dc9d640 5 monclient: authenticate success, global_id 4592 2024-03-25T18:24:46.363 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.359+0000 7fd24dc9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:24:46.380 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.379+0000 7fd2467fc640 1 -- 172.21.15.97:0/3375821700 <== mon.2 v1:172.21.15.97:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd23c0017d0 con 0x7fd24806ea40 2024-03-25T18:24:46.380 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.379+0000 7fd2467fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:46.380 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.379+0000 7fd2467fc640 1 -- 172.21.15.97:0/3375821700 <== mon.2 v1:172.21.15.97:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7fd23c001be0 con 0x7fd24806ea40 2024-03-25T18:24:46.381 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.379+0000 7fd2467fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:46.381 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.379+0000 7fd2467fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:24:46.381 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.379+0000 7fd2467fc640 10 monclient: dump: 2024-03-25T18:24:46.381 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:24:46.381 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:24:46.381 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:24:46.381 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:24:46.381 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:46.381 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:24:46.381 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:24:46.381 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:24:46.381 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:24:46.381 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:24:46.381 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.379+0000 7fd24dc9d640 10 monclient: get_monmap_and_config success 2024-03-25T18:24:46.382 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.379+0000 7fd24dc9d640 4 set_mon_vals no callback set 2024-03-25T18:24:46.383 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.379+0000 7fd24dc9d640 10 monclient: shutdown 2024-03-25T18:24:46.383 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.379+0000 7fd24dc9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:46.383 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.379+0000 7fd24dc9d640 1 -- 172.21.15.97:0/3375821700 >> v1:172.21.15.97:6790/0 conn(0x7fd24806ea40 legacy=0x7fd24813ee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:46.383 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.380+0000 7fd24dc9d640 1 -- 172.21.15.97:0/3375821700 shutdown_connections 2024-03-25T18:24:46.383 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.380+0000 7fd24dc9d640 1 -- 172.21.15.97:0/3375821700 >> 172.21.15.97:0/3375821700 conn(0x7fd24806b630 msgr2=0x7fd2480b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:46.383 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.380+0000 7fd24dc9d640 1 -- 172.21.15.97:0/3375821700 shutdown_connections 2024-03-25T18:24:46.383 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.380+0000 7fd24dc9d640 1 -- 172.21.15.97:0/3375821700 wait complete. 2024-03-25T18:24:46.383 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.380+0000 7fd24dc9d640 10 monclient: build_initial_monmap 2024-03-25T18:24:46.383 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.380+0000 7fd24dc9d640 10 monclient: monmap: 2024-03-25T18:24:46.383 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:24:46.384 INFO:teuthology.misc.health.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:24:46.384 INFO:teuthology.misc.health.smithi097.stderr:last_changed 0.000000 2024-03-25T18:24:46.384 INFO:teuthology.misc.health.smithi097.stderr:created 0.000000 2024-03-25T18:24:46.384 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:46.384 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 1 2024-03-25T18:24:46.384 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:24:46.384 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T18:24:46.384 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-c 2024-03-25T18:24:46.384 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:24:46.384 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.381+0000 7fd24dc9d640 1 librados: starting msgr at 2024-03-25T18:24:46.384 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.381+0000 7fd24dc9d640 1 librados: starting objecter 2024-03-25T18:24:46.384 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.381+0000 7fd24dc9d640 1 Processor -- start 2024-03-25T18:24:46.384 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.381+0000 7fd24dc9d640 1 -- start start 2024-03-25T18:24:46.384 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.381+0000 7fd24dc9d640 1 librados: setting wanted keys 2024-03-25T18:24:46.384 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.381+0000 7fd24dc9d640 1 librados: calling monclient init 2024-03-25T18:24:46.385 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.381+0000 7fd24dc9d640 10 monclient: init 2024-03-25T18:24:46.385 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.382+0000 7fd24dc9d640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:46.386 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.382+0000 7fd24dc9d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:24:46.386 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.382+0000 7fd24dc9d640 10 monclient(hunting): picked mon.noname-c con 0x7fd24806ea40 addr v1:172.21.15.97:6790/0 2024-03-25T18:24:46.386 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.382+0000 7fd24dc9d640 10 monclient(hunting): picked mon.noname-b con 0x7fd248071800 addr v1:172.21.15.98:6789/0 2024-03-25T18:24:46.386 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.382+0000 7fd24dc9d640 10 monclient(hunting): picked mon.noname-a con 0x7fd248071c40 addr v1:172.21.15.97:6789/0 2024-03-25T18:24:46.386 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.382+0000 7fd24dc9d640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd24811b9e0 con 0x7fd248071c40 2024-03-25T18:24:46.386 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.382+0000 7fd24dc9d640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd24813ab80 con 0x7fd248071800 2024-03-25T18:24:46.386 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.382+0000 7fd24dc9d640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd24813ad10 con 0x7fd24806ea40 2024-03-25T18:24:46.386 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.382+0000 7fd24dc9d640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:46.386 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.382+0000 7fd24dc9d640 10 monclient(hunting): authenticate will time out at 8098.658276s 2024-03-25T18:24:46.395 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.393+0000 7fd246ffd640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fd248071800 0x7fd248139920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.97:53380/0 (socket says 172.21.15.97:53380) 2024-03-25T18:24:46.395 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.393+0000 7fd246ffd640 1 -- 172.21.15.97:0/4224970002 learned_addr learned my addr 172.21.15.97:0/4224970002 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:24:46.395 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.394+0000 7fd22ffff640 10 client.?.objecter ms_handle_connect 0x7fd248071800 2024-03-25T18:24:46.395 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.394+0000 7fd22ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:46.398 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.397+0000 7fd22ffff640 1 -- 172.21.15.97:0/4224970002 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 445864128 0 0) 0x7fd24813ab80 con 0x7fd248071800 2024-03-25T18:24:46.398 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.397+0000 7fd22ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:24:46.398 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.397+0000 7fd22ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:46.398 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.397+0000 7fd22ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:46.399 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.397+0000 7fd22ffff640 10 monclient(hunting): my global_id is 4612 2024-03-25T18:24:46.399 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.397+0000 7fd22ffff640 1 -- 172.21.15.97:0/4224970002 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd234004eb0 con 0x7fd248071800 2024-03-25T18:24:46.404 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.402+0000 7fd22ffff640 10 client.?.objecter ms_handle_connect 0x7fd248071c40 2024-03-25T18:24:46.404 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.402+0000 7fd22ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:46.405 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.404+0000 7fd22ffff640 1 -- 172.21.15.97:0/4224970002 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 597476856 0 0) 0x7fd234004eb0 con 0x7fd248071800 2024-03-25T18:24:46.405 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.404+0000 7fd22ffff640 1 -- 172.21.15.97:0/4224970002 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd24813ab80 con 0x7fd248071800 2024-03-25T18:24:46.405 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.404+0000 7fd22ffff640 1 -- 172.21.15.97:0/4224970002 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7fd238001a10 con 0x7fd248071800 2024-03-25T18:24:46.406 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.404+0000 7fd22ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:46.406 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.404+0000 7fd22ffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:24:46.406 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.404+0000 7fd22ffff640 10 monclient(hunting): dump: 2024-03-25T18:24:46.406 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:24:46.406 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:24:46.406 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:24:46.406 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:24:46.406 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:46.406 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:24:46.406 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:24:46.406 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:24:46.406 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:24:46.406 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:24:46.407 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.404+0000 7fd22ffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:24:46.412 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.410+0000 7fd22ffff640 10 client.?.objecter ms_handle_connect 0x7fd24806ea40 2024-03-25T18:24:46.412 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.410+0000 7fd22ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:46.418 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.417+0000 7fd22ffff640 1 -- 172.21.15.97:0/4224970002 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1753646347 0 0) 0x7fd24811b9e0 con 0x7fd248071c40 2024-03-25T18:24:46.418 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.417+0000 7fd22ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:24:46.418 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.417+0000 7fd22ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:46.418 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.417+0000 7fd22ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:46.418 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.417+0000 7fd22ffff640 10 monclient(hunting): my global_id is 4605 2024-03-25T18:24:46.418 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.417+0000 7fd22ffff640 1 -- 172.21.15.97:0/4224970002 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd234004eb0 con 0x7fd248071c40 2024-03-25T18:24:46.420 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.418+0000 7fd22ffff640 1 -- 172.21.15.97:0/4224970002 <== mon.1 v1:172.21.15.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1188740675 0 0) 0x7fd24813ab80 con 0x7fd248071800 2024-03-25T18:24:46.420 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.418+0000 7fd22ffff640 1 -- 172.21.15.97:0/4224970002 >> v1:172.21.15.97:6790/0 conn(0x7fd24806ea40 legacy=0x7fd2481daba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:46.420 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.418+0000 7fd22ffff640 1 -- 172.21.15.97:0/4224970002 >> v1:172.21.15.97:6789/0 conn(0x7fd248071c40 legacy=0x7fd24811be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:46.421 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.418+0000 7fd22ffff640 10 monclient: _finish_hunting 0 2024-03-25T18:24:46.421 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.418+0000 7fd22ffff640 1 monclient: found mon.b 2024-03-25T18:24:46.421 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.418+0000 7fd22ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:46.421 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.418+0000 7fd22ffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:24:46.421 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.418+0000 7fd22ffff640 1 -- 172.21.15.97:0/4224970002 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2481db2b0 con 0x7fd248071800 2024-03-25T18:24:46.421 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.418+0000 7fd22ffff640 10 monclient: _finish_auth 0 2024-03-25T18:24:46.421 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.418+0000 7fd22ffff640 10 monclient: _check_auth_tickets 2024-03-25T18:24:46.421 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.418+0000 7fd22ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:46.421 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.418+0000 7fd24dc9d640 5 monclient: authenticate success, global_id 4612 2024-03-25T18:24:46.421 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.418+0000 7fd24dc9d640 10 monclient: _renew_subs 2024-03-25T18:24:46.421 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.418+0000 7fd24dc9d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:24:46.421 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.418+0000 7fd24dc9d640 1 -- 172.21.15.97:0/4224970002 --> v1:172.21.15.98:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2481db520 con 0x7fd248071800 2024-03-25T18:24:46.421 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.418+0000 7fd24dc9d640 10 client.4612.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:24:46.421 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.418+0000 7fd24dc9d640 10 monclient: _renew_subs 2024-03-25T18:24:46.422 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.418+0000 7fd24dc9d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:24:46.422 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.418+0000 7fd24dc9d640 1 -- 172.21.15.97:0/4224970002 --> v1:172.21.15.98:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd24813a210 con 0x7fd248071800 2024-03-25T18:24:46.422 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.418+0000 7fd24dc9d640 1 librados: init done 2024-03-25T18:24:46.422 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.419+0000 7fd24dc9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:46.422 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.419+0000 7fd24dc9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:46.423 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.419+0000 7fd24dc9d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:24:46.423 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.419+0000 7fd24dc9d640 1 -- 172.21.15.97:0/4224970002 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2481db760 con 0x7fd248071800 2024-03-25T18:24:46.696 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.694+0000 7fd22ffff640 1 -- 172.21.15.97:0/4224970002 <== mon.1 v1:172.21.15.98:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd238001cf0 con 0x7fd248071800 2024-03-25T18:24:46.696 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.694+0000 7fd22ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:46.696 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.694+0000 7fd22ffff640 1 -- 172.21.15.97:0/4224970002 <== mon.1 v1:172.21.15.98:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7fd238003a60 con 0x7fd248071800 2024-03-25T18:24:46.696 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.694+0000 7fd22ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:46.696 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.694+0000 7fd22ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:24:46.696 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.694+0000 7fd22ffff640 10 monclient: dump: 2024-03-25T18:24:46.696 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:24:46.696 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:24:46.697 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:24:46.697 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:24:46.697 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:46.697 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:24:46.697 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:24:46.697 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:24:46.697 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:24:46.697 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:24:46.697 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.694+0000 7fd245ffb640 4 set_mon_vals no callback set 2024-03-25T18:24:46.697 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.694+0000 7fd22ffff640 1 -- 172.21.15.97:0/4224970002 <== mon.1 v1:172.21.15.98:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2377622137 0 0) 0x7fd238003c80 con 0x7fd248071800 2024-03-25T18:24:46.698 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.695+0000 7fd2477fe640 10 monclient: get_auth_request con 0x7fd23406dd90 auth_method 0 2024-03-25T18:24:46.698 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.695+0000 7fd22ffff640 10 client.4612.objecter ms_handle_connect 0x7fd23406dd90 2024-03-25T18:24:46.701 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.699+0000 7fd22ffff640 1 -- 172.21.15.97:0/4224970002 <== mon.1 v1:172.21.15.98:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 430711918 0 0) 0x7fd238054890 con 0x7fd248071800 2024-03-25T18:24:46.701 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.699+0000 7fd22ffff640 10 client.4612.objecter ms_dispatch 0x7fd248000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:24:46.701 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.699+0000 7fd22ffff640 3 client.4612.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:24:46.701 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.699+0000 7fd22ffff640 3 client.4612.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:24:46.701 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.699+0000 7fd22ffff640 20 client.4612.objecter dump_active .. 0 homeless 2024-03-25T18:24:46.702 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.700+0000 7fd22ffff640 1 -- 172.21.15.97:0/4224970002 <== mon.1 v1:172.21.15.98:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd2380503a0 con 0x7fd248071800 2024-03-25T18:24:46.702 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.700+0000 7fd22ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:46.702 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.700+0000 7fd22ffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:24:46.956 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.954+0000 7fd24dc9d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:46.956 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.954+0000 7fd24dc9d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:46.956 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.954+0000 7fd24dc9d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:24:46.956 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.954+0000 7fd24dc9d640 1 -- 172.21.15.97:0/4224970002 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd2480627b0 con 0x7fd248071800 2024-03-25T18:24:46.975 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.973+0000 7fd22ffff640 1 -- 172.21.15.97:0/4224970002 <== mon.1 v1:172.21.15.98:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2169427412) 0x7fd2380522d0 con 0x7fd248071800 2024-03-25T18:24:46.975 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.973+0000 7fd22ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:46.975 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.973+0000 7fd22ffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:24:46.975 INFO:teuthology.misc.health.smithi097.stdout: 2024-03-25T18:24:46.976 INFO:teuthology.misc.health.smithi097.stdout:{"epoch":10,"fsid":"4c3d472c-79b8-486b-9e3d-2738d4306754","created":"2024-03-25T18:16:14.737309+0000","modified":"2024-03-25T18:16:20.357492+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:20.357492+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cb5152b9-262f-45c3-94b8-18d0e81df558","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6daf8793-0838-473d-a7d6-9a0d2894c75d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"136f1493-3b9c-4f43-a2e7-641482d69dae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"397d538c-4f64-4daf-9e8b-662d09f91327","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4f1df08-663f-4bda-93f8-3878ab7c24b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c4b18d70-a3ce-4b15-9b70-eae5525bd761","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8dd4033a-de0e-4f8a-96dc-59f3c1f825b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0077722e-ffa6-4a3d-9891-929d9869b286","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:24:46.978 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.976+0000 7fd24dc9d640 10 librados: watch_flush enter 2024-03-25T18:24:46.978 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.976+0000 7fd24dc9d640 10 librados: watch_flush exit 2024-03-25T18:24:46.978 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.976+0000 7fd24dc9d640 20 client.4612.objecter shutdown clearing up homeless session... 2024-03-25T18:24:46.978 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.976+0000 7fd24dc9d640 10 client.4612.objecter successfully canceled tick 2024-03-25T18:24:46.978 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.976+0000 7fd24dc9d640 1 -- 172.21.15.97:0/4224970002 >> v1:172.21.15.98:6800/800387573 conn(0x7fd23406dd90 legacy=0x7fd234070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:46.979 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.976+0000 7fd24dc9d640 10 monclient: shutdown 2024-03-25T18:24:46.979 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.976+0000 7fd24dc9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:46.979 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.976+0000 7fd24dc9d640 1 -- 172.21.15.97:0/4224970002 >> v1:172.21.15.98:6789/0 conn(0x7fd248071800 legacy=0x7fd248139920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:46.979 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.976+0000 7fd24dc9d640 1 -- 172.21.15.97:0/4224970002 shutdown_connections 2024-03-25T18:24:46.979 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.976+0000 7fd24dc9d640 1 -- 172.21.15.97:0/4224970002 >> 172.21.15.97:0/4224970002 conn(0x7fd24806b630 msgr2=0x7fd2480ae370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:46.979 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.976+0000 7fd24dc9d640 1 -- 172.21.15.97:0/4224970002 shutdown_connections 2024-03-25T18:24:46.979 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.976+0000 7fd24dc9d640 1 -- 172.21.15.97:0/4224970002 wait complete. 2024-03-25T18:24:46.979 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:46.976+0000 7fd24dc9d640 1 librados: shutdown 2024-03-25T18:24:46.991 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:24:53.797 DEBUG:teuthology.orchestra.run.smithi097:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:24:53.914 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:53.912+0000 7f5df230e640 10 monclient: get_monmap_and_config 2024-03-25T18:24:53.914 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:53.912+0000 7f5df230e640 10 monclient: build_initial_monmap 2024-03-25T18:24:53.914 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:53.912+0000 7f5df230e640 10 monclient: monmap: 2024-03-25T18:24:53.914 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:24:53.915 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:24:53.915 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:24:53.914161+0000 2024-03-25T18:24:53.915 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:24:53.914161+0000 2024-03-25T18:24:53.915 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:53.915 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:24:53.915 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:24:53.915 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T18:24:53.915 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-b 2024-03-25T18:24:53.915 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:24:53.917 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:53.913+0000 7f5df230e640 1 Processor -- start 2024-03-25T18:24:53.917 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:53.913+0000 7f5df230e640 1 -- start start 2024-03-25T18:24:53.917 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:53.913+0000 7f5df230e640 10 monclient: init 2024-03-25T18:24:53.917 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:53.914+0000 7f5df230e640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:53.917 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:53.914+0000 7f5df230e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:24:53.917 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:53.914+0000 7f5df230e640 10 monclient(hunting): picked mon.noname-c con 0x7f5dec0b4d60 addr v1:172.21.15.98:6789/0 2024-03-25T18:24:53.917 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:53.914+0000 7f5df230e640 10 monclient(hunting): picked mon.noname-b con 0x7f5dec0b1730 addr v1:172.21.15.97:6790/0 2024-03-25T18:24:53.917 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:53.914+0000 7f5df230e640 10 monclient(hunting): picked mon.noname-a con 0x7f5dec14be90 addr v1:172.21.15.97:6789/0 2024-03-25T18:24:53.917 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:53.914+0000 7f5df230e640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5dec0af940 con 0x7f5dec14be90 2024-03-25T18:24:53.917 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:53.914+0000 7f5df230e640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5dec0b0a70 con 0x7f5dec0b4d60 2024-03-25T18:24:53.917 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:53.914+0000 7f5df230e640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5dec0b0d50 con 0x7f5dec0b1730 2024-03-25T18:24:53.917 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:53.914+0000 7f5df230e640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:53.918 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:53.914+0000 7f5df230e640 10 monclient(hunting): authenticate will time out at 8106.190681s 2024-03-25T18:24:53.923 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:53.921+0000 7f5df08ab640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f5dec14be90 0x7f5dec14e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:52322/0 (socket says 172.21.15.97:52322) 2024-03-25T18:24:53.923 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:53.921+0000 7f5df08ab640 1 -- 172.21.15.97:0/3345695702 learned_addr learned my addr 172.21.15.97:0/3345695702 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:24:53.933 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:53.931+0000 7f5deaffd640 1 -- 172.21.15.97:0/3345695702 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2509461587 0 0) 0x7f5dec0b0a70 con 0x7f5dec0b4d60 2024-03-25T18:24:53.933 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:53.931+0000 7f5deaffd640 10 monclient(hunting): _negotiate 2024-03-25T18:24:53.933 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:53.931+0000 7f5deaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:53.933 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:53.931+0000 7f5deaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:53.933 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:53.931+0000 7f5deaffd640 10 monclient(hunting): my global_id is 4615 2024-03-25T18:24:53.933 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:53.932+0000 7f5deaffd640 1 -- 172.21.15.97:0/3345695702 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5dd4003520 con 0x7f5dec0b4d60 2024-03-25T18:24:53.942 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:53.940+0000 7f5deaffd640 1 -- 172.21.15.97:0/3345695702 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1391082410 0 0) 0x7f5dec0b0d50 con 0x7f5dec0b1730 2024-03-25T18:24:53.942 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:53.940+0000 7f5deaffd640 10 monclient(hunting): _negotiate 2024-03-25T18:24:53.942 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:53.940+0000 7f5deaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:53.942 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:53.940+0000 7f5deaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:53.942 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:53.940+0000 7f5deaffd640 10 monclient(hunting): my global_id is 4598 2024-03-25T18:24:53.942 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:53.940+0000 7f5deaffd640 1 -- 172.21.15.97:0/3345695702 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5dec0b0a70 con 0x7f5dec0b1730 2024-03-25T18:24:53.943 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:53.941+0000 7f5deaffd640 1 -- 172.21.15.97:0/3345695702 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2425037049 0 0) 0x7f5dec0af940 con 0x7f5dec14be90 2024-03-25T18:24:53.943 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:53.941+0000 7f5deaffd640 10 monclient(hunting): _negotiate 2024-03-25T18:24:53.943 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:53.941+0000 7f5deaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:53.943 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:53.941+0000 7f5deaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:53.943 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:53.941+0000 7f5deaffd640 10 monclient(hunting): my global_id is 4608 2024-03-25T18:24:53.943 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:53.941+0000 7f5deaffd640 1 -- 172.21.15.97:0/3345695702 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5dec0b0d50 con 0x7f5dec14be90 2024-03-25T18:24:53.952 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:53.951+0000 7f5deaffd640 1 -- 172.21.15.97:0/3345695702 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3871278792 0 0) 0x7f5dd4003520 con 0x7f5dec0b4d60 2024-03-25T18:24:53.953 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:53.951+0000 7f5deaffd640 1 -- 172.21.15.97:0/3345695702 >> v1:172.21.15.97:6790/0 conn(0x7f5dec0b1730 legacy=0x7f5dec0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:53.953 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:53.951+0000 7f5deaffd640 1 -- 172.21.15.97:0/3345695702 >> v1:172.21.15.97:6789/0 conn(0x7f5dec14be90 legacy=0x7f5dec14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:53.953 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:53.951+0000 7f5deaffd640 10 monclient: _finish_hunting 0 2024-03-25T18:24:53.953 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:53.951+0000 7f5deaffd640 1 monclient: found mon.noname-c 2024-03-25T18:24:53.953 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:53.951+0000 7f5deaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:53.953 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:53.951+0000 7f5deaffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.98:6789/0 2024-03-25T18:24:53.953 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:53.951+0000 7f5deaffd640 1 -- 172.21.15.97:0/3345695702 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dec0b1030 con 0x7f5dec0b4d60 2024-03-25T18:24:53.953 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:53.951+0000 7f5deaffd640 10 monclient: _finish_auth 0 2024-03-25T18:24:53.953 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:53.951+0000 7f5deaffd640 10 monclient: _check_auth_tickets 2024-03-25T18:24:53.953 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:53.951+0000 7f5deaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:53.953 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:53.951+0000 7f5deaffd640 1 -- 172.21.15.97:0/3345695702 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f5de0001690 con 0x7f5dec0b4d60 2024-03-25T18:24:53.953 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:53.951+0000 7f5deaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:53.953 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:53.951+0000 7f5deaffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:24:53.954 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:53.951+0000 7f5deaffd640 10 monclient: dump: 2024-03-25T18:24:53.954 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:24:53.954 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:24:53.954 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:24:53.954 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:24:53.954 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:53.954 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:24:53.954 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:24:53.954 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:24:53.954 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:24:53.954 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:24:53.954 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:53.951+0000 7f5df230e640 5 monclient: authenticate success, global_id 4615 2024-03-25T18:24:53.954 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:53.951+0000 7f5df230e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:24:53.968 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:53.966+0000 7f5deaffd640 1 -- 172.21.15.97:0/3345695702 <== mon.1 v1:172.21.15.98:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5de00017f0 con 0x7f5dec0b4d60 2024-03-25T18:24:53.968 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:53.966+0000 7f5deaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:53.968 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:53.966+0000 7f5deaffd640 1 -- 172.21.15.97:0/3345695702 <== mon.1 v1:172.21.15.98:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f5de0001c00 con 0x7f5dec0b4d60 2024-03-25T18:24:53.968 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:53.966+0000 7f5df230e640 10 monclient: get_monmap_and_config success 2024-03-25T18:24:53.968 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:53.966+0000 7f5df230e640 4 set_mon_vals no callback set 2024-03-25T18:24:53.968 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:53.966+0000 7f5deaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:53.968 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:53.966+0000 7f5deaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:24:53.968 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:53.966+0000 7f5deaffd640 10 monclient: dump: 2024-03-25T18:24:53.968 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:24:53.968 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:24:53.968 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:24:53.968 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:24:53.968 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:53.969 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:24:53.969 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:24:53.969 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:24:53.969 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:24:53.969 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:24:53.971 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:53.967+0000 7f5df230e640 10 monclient: shutdown 2024-03-25T18:24:53.971 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:53.967+0000 7f5df230e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:53.971 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:53.967+0000 7f5df230e640 1 -- 172.21.15.97:0/3345695702 >> v1:172.21.15.98:6789/0 conn(0x7f5dec0b4d60 legacy=0x7f5dec13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:53.971 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:53.967+0000 7f5df230e640 1 -- 172.21.15.97:0/3345695702 shutdown_connections 2024-03-25T18:24:53.971 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:53.967+0000 7f5df230e640 1 -- 172.21.15.97:0/3345695702 >> 172.21.15.97:0/3345695702 conn(0x7f5dec0617a0 msgr2=0x7f5dec061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:53.971 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:53.967+0000 7f5df230e640 1 -- 172.21.15.97:0/3345695702 shutdown_connections 2024-03-25T18:24:53.971 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:53.967+0000 7f5df230e640 1 -- 172.21.15.97:0/3345695702 wait complete. 2024-03-25T18:24:53.971 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:53.968+0000 7f5df230e640 10 monclient: build_initial_monmap 2024-03-25T18:24:53.971 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:53.968+0000 7f5df230e640 10 monclient: monmap: 2024-03-25T18:24:53.972 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:24:53.972 INFO:teuthology.misc.health.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:24:53.972 INFO:teuthology.misc.health.smithi097.stderr:last_changed 0.000000 2024-03-25T18:24:53.972 INFO:teuthology.misc.health.smithi097.stderr:created 0.000000 2024-03-25T18:24:53.972 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:53.972 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 1 2024-03-25T18:24:53.972 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:24:53.972 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T18:24:53.972 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-c 2024-03-25T18:24:53.972 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:24:53.972 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:53.968+0000 7f5df230e640 1 librados: starting msgr at 2024-03-25T18:24:53.972 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:53.968+0000 7f5df230e640 1 librados: starting objecter 2024-03-25T18:24:53.972 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:53.968+0000 7f5df230e640 1 Processor -- start 2024-03-25T18:24:53.972 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:53.968+0000 7f5df230e640 1 -- start start 2024-03-25T18:24:53.972 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:53.968+0000 7f5df230e640 1 librados: setting wanted keys 2024-03-25T18:24:53.973 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:53.968+0000 7f5df230e640 1 librados: calling monclient init 2024-03-25T18:24:53.973 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:53.968+0000 7f5df230e640 10 monclient: init 2024-03-25T18:24:53.973 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:53.969+0000 7f5df230e640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:53.973 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:53.969+0000 7f5df230e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:24:53.973 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:53.969+0000 7f5df230e640 10 monclient(hunting): picked mon.noname-c con 0x7f5dec0b1730 addr v1:172.21.15.97:6790/0 2024-03-25T18:24:53.973 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:53.969+0000 7f5df230e640 10 monclient(hunting): picked mon.noname-a con 0x7f5dec0b4d60 addr v1:172.21.15.97:6789/0 2024-03-25T18:24:53.973 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:53.969+0000 7f5df230e640 10 monclient(hunting): picked mon.noname-b con 0x7f5dec14be90 addr v1:172.21.15.98:6789/0 2024-03-25T18:24:53.973 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:53.969+0000 7f5df230e640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5dec140ce0 con 0x7f5dec0b4d60 2024-03-25T18:24:53.973 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:53.969+0000 7f5df230e640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5dec1d7a00 con 0x7f5dec14be90 2024-03-25T18:24:53.973 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:53.969+0000 7f5df230e640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5dec1d85a0 con 0x7f5dec0b1730 2024-03-25T18:24:53.973 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:53.969+0000 7f5df230e640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:53.973 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:53.969+0000 7f5df230e640 10 monclient(hunting): authenticate will time out at 8106.245507s 2024-03-25T18:24:53.976 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:53.975+0000 7f5debfff640 1 --1- >> v1:172.21.15.97:6790/0 conn(0x7f5dec0b1730 0x7f5dec1d7e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.97:6790/0 says I am v1:172.21.15.97:40924/0 (socket says 172.21.15.97:40924) 2024-03-25T18:24:53.976 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:53.975+0000 7f5debfff640 1 -- 172.21.15.97:0/3311780655 learned_addr learned my addr 172.21.15.97:0/3311780655 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:24:53.977 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:53.976+0000 7f5de8ff9640 10 client.?.objecter ms_handle_connect 0x7f5dec0b1730 2024-03-25T18:24:53.977 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:53.976+0000 7f5de8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:53.989 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:53.987+0000 7f5de8ff9640 1 -- 172.21.15.97:0/3311780655 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 512227472 0 0) 0x7f5dec1d85a0 con 0x7f5dec0b1730 2024-03-25T18:24:53.989 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:53.987+0000 7f5de8ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:24:53.989 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:53.987+0000 7f5de8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:53.989 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:53.987+0000 7f5de8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:53.989 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:53.987+0000 7f5de8ff9640 10 monclient(hunting): my global_id is 4601 2024-03-25T18:24:53.989 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:53.987+0000 7f5de8ff9640 1 -- 172.21.15.97:0/3311780655 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5dd8004eb0 con 0x7f5dec0b1730 2024-03-25T18:24:53.995 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:53.993+0000 7f5de8ff9640 10 client.?.objecter ms_handle_connect 0x7f5dec0b4d60 2024-03-25T18:24:53.995 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:53.993+0000 7f5de8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:53.995 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:53.994+0000 7f5de8ff9640 10 client.?.objecter ms_handle_connect 0x7f5dec14be90 2024-03-25T18:24:53.995 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:53.994+0000 7f5de8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:54.007 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:54.006+0000 7f5de8ff9640 1 -- 172.21.15.97:0/3311780655 <== mon.2 v1:172.21.15.97:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2957713154 0 0) 0x7f5dd8004eb0 con 0x7f5dec0b1730 2024-03-25T18:24:54.008 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:54.006+0000 7f5de8ff9640 1 -- 172.21.15.97:0/3311780655 --> v1:172.21.15.97:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5dec1d85a0 con 0x7f5dec0b1730 2024-03-25T18:24:54.008 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:54.006+0000 7f5de8ff9640 1 -- 172.21.15.97:0/3311780655 <== mon.2 v1:172.21.15.97:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f5de0001bc0 con 0x7f5dec0b1730 2024-03-25T18:24:54.008 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:54.006+0000 7f5de8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:54.008 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:54.006+0000 7f5de8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:24:54.008 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:54.006+0000 7f5de8ff9640 10 monclient(hunting): dump: 2024-03-25T18:24:54.008 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:24:54.008 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:24:54.008 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:24:54.008 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:24:54.008 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:54.008 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:24:54.008 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:24:54.008 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:24:54.009 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:24:54.009 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:24:54.009 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:54.006+0000 7f5de8ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:24:54.011 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:54.009+0000 7f5de8ff9640 1 -- 172.21.15.97:0/3311780655 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 522690009 0 0) 0x7f5dec0b0a70 con 0x7f5dec0b4d60 2024-03-25T18:24:54.011 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:54.009+0000 7f5de8ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:24:54.011 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:54.009+0000 7f5de8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:54.011 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:54.009+0000 7f5de8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:54.011 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:54.009+0000 7f5de8ff9640 10 monclient(hunting): my global_id is 4611 2024-03-25T18:24:54.011 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:54.009+0000 7f5de8ff9640 1 -- 172.21.15.97:0/3311780655 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5dd8004eb0 con 0x7f5dec0b4d60 2024-03-25T18:24:54.018 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:54.016+0000 7f5de8ff9640 1 -- 172.21.15.97:0/3311780655 <== mon.2 v1:172.21.15.97:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2729244782 0 0) 0x7f5dec1d85a0 con 0x7f5dec0b1730 2024-03-25T18:24:54.018 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:54.016+0000 7f5de8ff9640 1 -- 172.21.15.97:0/3311780655 >> v1:172.21.15.98:6789/0 conn(0x7f5dec14be90 legacy=0x7f5dec142cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:54.018 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:54.016+0000 7f5de8ff9640 1 -- 172.21.15.97:0/3311780655 >> v1:172.21.15.97:6789/0 conn(0x7f5dec0b4d60 legacy=0x7f5dec13b5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:54.018 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:54.016+0000 7f5de8ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:24:54.018 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:54.016+0000 7f5de8ff9640 1 monclient: found mon.c 2024-03-25T18:24:54.018 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:54.016+0000 7f5de8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:54.018 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:54.016+0000 7f5de8ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.97:6790/0 2024-03-25T18:24:54.019 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:54.016+0000 7f5de8ff9640 1 -- 172.21.15.97:0/3311780655 --> v1:172.21.15.97:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dec1d7690 con 0x7f5dec0b1730 2024-03-25T18:24:54.019 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:54.016+0000 7f5de8ff9640 10 monclient: _finish_auth 0 2024-03-25T18:24:54.019 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:54.016+0000 7f5de8ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:24:54.019 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:54.016+0000 7f5de8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:54.019 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:54.016+0000 7f5df230e640 5 monclient: authenticate success, global_id 4601 2024-03-25T18:24:54.019 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:54.016+0000 7f5df230e640 10 monclient: _renew_subs 2024-03-25T18:24:54.019 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:54.016+0000 7f5df230e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.97:6790/0 2024-03-25T18:24:54.019 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:54.016+0000 7f5df230e640 1 -- 172.21.15.97:0/3311780655 --> v1:172.21.15.97:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5dec13bcc0 con 0x7f5dec0b1730 2024-03-25T18:24:54.019 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:54.016+0000 7f5df230e640 10 client.4601.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:24:54.019 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:54.016+0000 7f5df230e640 10 monclient: _renew_subs 2024-03-25T18:24:54.019 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:54.016+0000 7f5df230e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.97:6790/0 2024-03-25T18:24:54.019 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:54.016+0000 7f5df230e640 1 -- 172.21.15.97:0/3311780655 --> v1:172.21.15.97:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5dec1405f0 con 0x7f5dec0b1730 2024-03-25T18:24:54.019 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:54.016+0000 7f5df230e640 1 librados: init done 2024-03-25T18:24:54.020 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:54.017+0000 7f5df230e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:54.020 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:54.017+0000 7f5df230e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:54.020 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:54.017+0000 7f5df230e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.97:6790/0 2024-03-25T18:24:54.020 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:54.017+0000 7f5df230e640 1 -- 172.21.15.97:0/3311780655 --> v1:172.21.15.97:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5dec0627b0 con 0x7f5dec0b1730 2024-03-25T18:24:54.312 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:54.311+0000 7f5de8ff9640 1 -- 172.21.15.97:0/3311780655 <== mon.2 v1:172.21.15.97:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5de00011f0 con 0x7f5dec0b1730 2024-03-25T18:24:54.313 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:54.311+0000 7f5de8ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:54.313 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:54.311+0000 7f5de8ff9640 1 -- 172.21.15.97:0/3311780655 <== mon.2 v1:172.21.15.97:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f5de0003b00 con 0x7f5dec0b1730 2024-03-25T18:24:54.313 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:54.311+0000 7f5de8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:54.313 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:54.311+0000 7f5de8ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:24:54.313 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:54.311+0000 7f5de8ff9640 10 monclient: dump: 2024-03-25T18:24:54.313 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:24:54.313 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:24:54.313 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:24:54.313 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:24:54.313 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:54.314 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:24:54.314 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:24:54.314 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:24:54.314 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:24:54.314 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:24:54.314 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:54.311+0000 7f5dea7fc640 4 set_mon_vals no callback set 2024-03-25T18:24:54.315 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:54.312+0000 7f5de8ff9640 1 -- 172.21.15.97:0/3311780655 <== mon.2 v1:172.21.15.97:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2377622137 0 0) 0x7f5de0003d20 con 0x7f5dec0b1730 2024-03-25T18:24:54.315 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:54.312+0000 7f5deb7fe640 10 monclient: get_auth_request con 0x7f5dd806dd40 auth_method 0 2024-03-25T18:24:54.315 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:54.312+0000 7f5de8ff9640 10 client.4601.objecter ms_handle_connect 0x7f5dd806dd40 2024-03-25T18:24:54.320 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:54.318+0000 7f5de8ff9640 1 -- 172.21.15.97:0/3311780655 <== mon.2 v1:172.21.15.97:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 430711918 0 0) 0x7f5de00548c0 con 0x7f5dec0b1730 2024-03-25T18:24:54.320 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:54.318+0000 7f5de8ff9640 10 client.4601.objecter ms_dispatch 0x7f5dec000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:24:54.320 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:54.318+0000 7f5de8ff9640 3 client.4601.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:24:54.320 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:54.318+0000 7f5de8ff9640 3 client.4601.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:24:54.320 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:54.319+0000 7f5de8ff9640 20 client.4601.objecter dump_active .. 0 homeless 2024-03-25T18:24:54.321 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:54.319+0000 7f5de8ff9640 1 -- 172.21.15.97:0/3311780655 <== mon.2 v1:172.21.15.97:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5de0050200 con 0x7f5dec0b1730 2024-03-25T18:24:54.321 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:54.319+0000 7f5de8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:54.321 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:54.319+0000 7f5de8ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:24:54.564 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:54.563+0000 7f5df230e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:54.564 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:54.563+0000 7f5df230e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:54.564 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:54.563+0000 7f5df230e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.97:6790/0 2024-03-25T18:24:54.565 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:54.563+0000 7f5df230e640 1 -- 172.21.15.97:0/3311780655 --> v1:172.21.15.97:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5dec069f40 con 0x7f5dec0b1730 2024-03-25T18:24:54.687 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:54.685+0000 7f5de8ff9640 1 -- 172.21.15.97:0/3311780655 <== mon.2 v1:172.21.15.97:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2169427412) 0x7f5de0052300 con 0x7f5dec0b1730 2024-03-25T18:24:54.687 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:54.685+0000 7f5de8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:54.687 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:54.685+0000 7f5de8ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:24:54.687 INFO:teuthology.misc.health.smithi097.stdout: 2024-03-25T18:24:54.687 INFO:teuthology.misc.health.smithi097.stdout:{"epoch":10,"fsid":"4c3d472c-79b8-486b-9e3d-2738d4306754","created":"2024-03-25T18:16:14.737309+0000","modified":"2024-03-25T18:16:20.357492+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:20.357492+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cb5152b9-262f-45c3-94b8-18d0e81df558","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6daf8793-0838-473d-a7d6-9a0d2894c75d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"136f1493-3b9c-4f43-a2e7-641482d69dae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"397d538c-4f64-4daf-9e8b-662d09f91327","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4f1df08-663f-4bda-93f8-3878ab7c24b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c4b18d70-a3ce-4b15-9b70-eae5525bd761","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8dd4033a-de0e-4f8a-96dc-59f3c1f825b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0077722e-ffa6-4a3d-9891-929d9869b286","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:24:54.689 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:54.687+0000 7f5df230e640 10 librados: watch_flush enter 2024-03-25T18:24:54.690 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:54.687+0000 7f5df230e640 10 librados: watch_flush exit 2024-03-25T18:24:54.690 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:54.687+0000 7f5df230e640 20 client.4601.objecter shutdown clearing up homeless session... 2024-03-25T18:24:54.690 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:54.687+0000 7f5df230e640 10 client.4601.objecter successfully canceled tick 2024-03-25T18:24:54.690 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:54.687+0000 7f5df230e640 1 -- 172.21.15.97:0/3311780655 >> v1:172.21.15.98:6800/800387573 conn(0x7f5dd806dd40 legacy=0x7f5dd8070200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:54.690 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:54.687+0000 7f5df230e640 10 monclient: shutdown 2024-03-25T18:24:54.690 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:54.687+0000 7f5df230e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:54.690 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:54.687+0000 7f5df230e640 1 -- 172.21.15.97:0/3311780655 >> v1:172.21.15.97:6790/0 conn(0x7f5dec0b1730 legacy=0x7f5dec1d7e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:54.690 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:54.687+0000 7f5df230e640 1 -- 172.21.15.97:0/3311780655 shutdown_connections 2024-03-25T18:24:54.690 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:54.687+0000 7f5df230e640 1 -- 172.21.15.97:0/3311780655 >> 172.21.15.97:0/3311780655 conn(0x7f5dec0617a0 msgr2=0x7f5dec061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:54.690 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:54.688+0000 7f5df230e640 1 -- 172.21.15.97:0/3311780655 shutdown_connections 2024-03-25T18:24:54.690 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:54.688+0000 7f5df230e640 1 -- 172.21.15.97:0/3311780655 wait complete. 2024-03-25T18:24:54.690 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:24:54.688+0000 7f5df230e640 1 librados: shutdown 2024-03-25T18:24:54.700 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:24:58.528 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:09:00.566 34931== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:58.534 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:09:00.572 34931== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:58.563 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:09:00.601 34931== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:58.583 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:09:00.622 34935== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:58.584 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:09:00.627 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:58.586 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:09:00.625 34935== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:58.587 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:09:00.630 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:58.616 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:09:00.659 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:58.618 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:09:00.656 34935== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:01.506 DEBUG:teuthology.orchestra.run.smithi097:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:25:01.625 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:01.623+0000 7f92a89b6640 10 monclient: get_monmap_and_config 2024-03-25T18:25:01.625 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:01.623+0000 7f92a89b6640 10 monclient: build_initial_monmap 2024-03-25T18:25:01.625 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:01.623+0000 7f92a89b6640 10 monclient: monmap: 2024-03-25T18:25:01.625 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:25:01.625 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:25:01.625 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:25:01.625056+0000 2024-03-25T18:25:01.625 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:25:01.625056+0000 2024-03-25T18:25:01.626 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:01.626 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:25:01.626 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:25:01.626 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T18:25:01.626 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-b 2024-03-25T18:25:01.626 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:25:01.627 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:01.624+0000 7f92a89b6640 1 Processor -- start 2024-03-25T18:25:01.627 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:01.624+0000 7f92a89b6640 1 -- start start 2024-03-25T18:25:01.627 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:01.624+0000 7f92a89b6640 10 monclient: init 2024-03-25T18:25:01.627 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:01.625+0000 7f92a89b6640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:01.628 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:01.625+0000 7f92a89b6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:25:01.628 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:01.625+0000 7f92a89b6640 10 monclient(hunting): picked mon.noname-b con 0x7f92a40b4d60 addr v1:172.21.15.97:6790/0 2024-03-25T18:25:01.628 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:01.625+0000 7f92a89b6640 10 monclient(hunting): picked mon.noname-c con 0x7f92a40b1730 addr v1:172.21.15.98:6789/0 2024-03-25T18:25:01.628 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:01.625+0000 7f92a89b6640 10 monclient(hunting): picked mon.noname-a con 0x7f92a414be90 addr v1:172.21.15.97:6789/0 2024-03-25T18:25:01.628 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:01.625+0000 7f92a89b6640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92a40af940 con 0x7f92a414be90 2024-03-25T18:25:01.628 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:01.625+0000 7f92a89b6640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92a40b0a70 con 0x7f92a40b1730 2024-03-25T18:25:01.628 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:01.625+0000 7f92a89b6640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92a40b0d50 con 0x7f92a40b4d60 2024-03-25T18:25:01.628 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:01.625+0000 7f92a89b6640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:01.628 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:01.625+0000 7f92a89b6640 10 monclient(hunting): authenticate will time out at 8113.901030s 2024-03-25T18:25:01.628 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:01.625+0000 7f92a1d9b640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f92a40b1730 0x7f92a40b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.97:42982/0 (socket says 172.21.15.97:42982) 2024-03-25T18:25:01.628 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:01.625+0000 7f92a1d9b640 1 -- 172.21.15.97:0/1525050356 learned_addr learned my addr 172.21.15.97:0/1525050356 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:25:01.688 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:01.686+0000 7f92a159a640 1 -- 172.21.15.97:0/1525050356 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 272169017 0 0) 0x7f92a40af940 con 0x7f92a414be90 2024-03-25T18:25:01.688 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:01.686+0000 7f92a159a640 10 monclient(hunting): _negotiate 2024-03-25T18:25:01.688 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:01.686+0000 7f92a159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:01.688 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:01.686+0000 7f92a159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:01.688 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:01.686+0000 7f92a159a640 10 monclient(hunting): my global_id is 4614 2024-03-25T18:25:01.688 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:01.686+0000 7f92a159a640 1 -- 172.21.15.97:0/1525050356 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f928c003520 con 0x7f92a414be90 2024-03-25T18:25:01.705 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:01.703+0000 7f92a159a640 1 -- 172.21.15.97:0/1525050356 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 405675397 0 0) 0x7f928c003520 con 0x7f92a414be90 2024-03-25T18:25:01.705 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:01.703+0000 7f92a159a640 1 -- 172.21.15.97:0/1525050356 >> v1:172.21.15.97:6790/0 conn(0x7f92a40b4d60 legacy=0x7f92a413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:01.705 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:01.703+0000 7f92a159a640 1 -- 172.21.15.97:0/1525050356 >> v1:172.21.15.98:6789/0 conn(0x7f92a40b1730 legacy=0x7f92a40b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:01.705 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:01.703+0000 7f92a159a640 10 monclient: _finish_hunting 0 2024-03-25T18:25:01.705 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:01.703+0000 7f92a159a640 1 monclient: found mon.noname-a 2024-03-25T18:25:01.706 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:01.703+0000 7f92a159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:01.706 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:01.703+0000 7f92a159a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.97:6789/0 2024-03-25T18:25:01.706 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:01.703+0000 7f92a159a640 1 -- 172.21.15.97:0/1525050356 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92a40b1030 con 0x7f92a414be90 2024-03-25T18:25:01.706 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:01.703+0000 7f92a159a640 10 monclient: _finish_auth 0 2024-03-25T18:25:01.706 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:01.703+0000 7f92a159a640 10 monclient: _check_auth_tickets 2024-03-25T18:25:01.706 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:01.703+0000 7f92a159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:01.706 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:01.703+0000 7f92a159a640 1 -- 172.21.15.97:0/1525050356 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f9294001690 con 0x7f92a414be90 2024-03-25T18:25:01.706 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:01.703+0000 7f92a159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:01.706 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:01.703+0000 7f92a159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:25:01.706 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:01.703+0000 7f92a159a640 10 monclient: dump: 2024-03-25T18:25:01.706 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:25:01.706 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:25:01.706 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:25:01.707 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:25:01.707 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:01.707 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:25:01.707 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:25:01.707 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:25:01.707 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:25:01.707 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:25:01.707 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:01.703+0000 7f92a89b6640 5 monclient: authenticate success, global_id 4614 2024-03-25T18:25:01.707 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:01.703+0000 7f92a89b6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:25:01.728 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:01.726+0000 7f92a159a640 1 -- 172.21.15.97:0/1525050356 <== mon.0 v1:172.21.15.97:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f92940017f0 con 0x7f92a414be90 2024-03-25T18:25:01.728 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:01.726+0000 7f92a159a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:01.728 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:01.726+0000 7f92a159a640 1 -- 172.21.15.97:0/1525050356 <== mon.0 v1:172.21.15.97:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f9294001c00 con 0x7f92a414be90 2024-03-25T18:25:01.728 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:01.726+0000 7f92a89b6640 10 monclient: get_monmap_and_config success 2024-03-25T18:25:01.728 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:01.726+0000 7f92a89b6640 4 set_mon_vals no callback set 2024-03-25T18:25:01.728 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:01.726+0000 7f92a159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:01.728 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:01.726+0000 7f92a159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:25:01.728 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:01.726+0000 7f92a159a640 10 monclient: dump: 2024-03-25T18:25:01.728 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:25:01.728 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:25:01.728 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:25:01.729 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:25:01.729 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:01.729 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:25:01.729 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:25:01.729 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:25:01.729 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:25:01.729 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:25:01.731 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:01.727+0000 7f92a89b6640 10 monclient: shutdown 2024-03-25T18:25:01.731 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:01.727+0000 7f92a89b6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:01.731 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:01.727+0000 7f92a89b6640 1 -- 172.21.15.97:0/1525050356 >> v1:172.21.15.97:6789/0 conn(0x7f92a414be90 legacy=0x7f92a414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:01.731 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:01.727+0000 7f92a89b6640 1 -- 172.21.15.97:0/1525050356 shutdown_connections 2024-03-25T18:25:01.731 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:01.727+0000 7f92a89b6640 1 -- 172.21.15.97:0/1525050356 >> 172.21.15.97:0/1525050356 conn(0x7f92a40617a0 msgr2=0x7f92a4061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:01.731 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:01.727+0000 7f92a89b6640 1 -- 172.21.15.97:0/1525050356 shutdown_connections 2024-03-25T18:25:01.731 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:01.727+0000 7f92a89b6640 1 -- 172.21.15.97:0/1525050356 wait complete. 2024-03-25T18:25:01.731 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:01.728+0000 7f92a89b6640 10 monclient: build_initial_monmap 2024-03-25T18:25:01.732 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:01.728+0000 7f92a89b6640 10 monclient: monmap: 2024-03-25T18:25:01.732 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:25:01.732 INFO:teuthology.misc.health.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:25:01.732 INFO:teuthology.misc.health.smithi097.stderr:last_changed 0.000000 2024-03-25T18:25:01.732 INFO:teuthology.misc.health.smithi097.stderr:created 0.000000 2024-03-25T18:25:01.732 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:01.732 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 1 2024-03-25T18:25:01.732 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:25:01.732 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T18:25:01.732 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-c 2024-03-25T18:25:01.732 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:25:01.732 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:01.728+0000 7f92a89b6640 1 librados: starting msgr at 2024-03-25T18:25:01.732 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:01.728+0000 7f92a89b6640 1 librados: starting objecter 2024-03-25T18:25:01.732 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:01.728+0000 7f92a89b6640 1 Processor -- start 2024-03-25T18:25:01.732 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:01.728+0000 7f92a89b6640 1 -- start start 2024-03-25T18:25:01.733 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:01.728+0000 7f92a89b6640 1 librados: setting wanted keys 2024-03-25T18:25:01.733 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:01.728+0000 7f92a89b6640 1 librados: calling monclient init 2024-03-25T18:25:01.733 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:01.728+0000 7f92a89b6640 10 monclient: init 2024-03-25T18:25:01.733 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:01.729+0000 7f92a89b6640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:01.733 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:01.729+0000 7f92a89b6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:25:01.733 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:01.729+0000 7f92a89b6640 10 monclient(hunting): picked mon.noname-b con 0x7f92a40b1730 addr v1:172.21.15.98:6789/0 2024-03-25T18:25:01.733 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:01.729+0000 7f92a89b6640 10 monclient(hunting): picked mon.noname-c con 0x7f92a40b4d60 addr v1:172.21.15.97:6790/0 2024-03-25T18:25:01.733 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:01.729+0000 7f92a89b6640 10 monclient(hunting): picked mon.noname-a con 0x7f92a414be90 addr v1:172.21.15.97:6789/0 2024-03-25T18:25:01.733 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:01.729+0000 7f92a89b6640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92a413bd10 con 0x7f92a414be90 2024-03-25T18:25:01.733 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:01.729+0000 7f92a89b6640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92a41dadb0 con 0x7f92a40b1730 2024-03-25T18:25:01.733 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:01.729+0000 7f92a89b6640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92a4138950 con 0x7f92a40b4d60 2024-03-25T18:25:01.733 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:01.729+0000 7f92a89b6640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:01.733 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:01.729+0000 7f92a89b6640 10 monclient(hunting): authenticate will time out at 8114.005347s 2024-03-25T18:25:01.742 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:01.741+0000 7f92a259c640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f92a40b1730 0x7f92a4138240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.97:43020/0 (socket says 172.21.15.97:43020) 2024-03-25T18:25:01.742 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:01.741+0000 7f92a259c640 1 -- 172.21.15.97:0/3764308665 learned_addr learned my addr 172.21.15.97:0/3764308665 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:25:01.745 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:01.743+0000 7f9292ffd640 10 client.?.objecter ms_handle_connect 0x7f92a40b1730 2024-03-25T18:25:01.745 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:01.743+0000 7f9292ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:01.751 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:01.749+0000 7f9292ffd640 1 -- 172.21.15.97:0/3764308665 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2021294697 0 0) 0x7f92a41dadb0 con 0x7f92a40b1730 2024-03-25T18:25:01.751 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:01.749+0000 7f9292ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:25:01.751 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:01.749+0000 7f9292ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:01.751 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:01.749+0000 7f9292ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:01.751 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:01.749+0000 7f9292ffd640 10 monclient(hunting): my global_id is 4624 2024-03-25T18:25:01.751 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:01.749+0000 7f9292ffd640 1 -- 172.21.15.97:0/3764308665 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9284004eb0 con 0x7f92a40b1730 2024-03-25T18:25:01.752 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:01.750+0000 7f9292ffd640 10 client.?.objecter ms_handle_connect 0x7f92a40b4d60 2024-03-25T18:25:01.752 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:01.750+0000 7f9292ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:01.753 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:01.752+0000 7f9292ffd640 10 client.?.objecter ms_handle_connect 0x7f92a414be90 2024-03-25T18:25:01.753 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:01.752+0000 7f9292ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:01.758 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:01.756+0000 7f9292ffd640 1 -- 172.21.15.97:0/3764308665 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2889273200 0 0) 0x7f92a4138950 con 0x7f92a40b4d60 2024-03-25T18:25:01.758 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:01.756+0000 7f9292ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:25:01.758 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:01.756+0000 7f9292ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:01.758 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:01.756+0000 7f9292ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:01.758 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:01.756+0000 7f9292ffd640 10 monclient(hunting): my global_id is 4607 2024-03-25T18:25:01.758 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:01.756+0000 7f9292ffd640 1 -- 172.21.15.97:0/3764308665 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92a41dadb0 con 0x7f92a40b4d60 2024-03-25T18:25:01.764 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:01.763+0000 7f9292ffd640 1 -- 172.21.15.97:0/3764308665 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3975348581 0 0) 0x7f92a413bd10 con 0x7f92a414be90 2024-03-25T18:25:01.764 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:01.763+0000 7f9292ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:25:01.764 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:01.763+0000 7f9292ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:01.765 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:01.763+0000 7f9292ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:01.765 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:01.763+0000 7f9292ffd640 10 monclient(hunting): my global_id is 4617 2024-03-25T18:25:01.765 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:01.763+0000 7f9292ffd640 1 -- 172.21.15.97:0/3764308665 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92a4138950 con 0x7f92a414be90 2024-03-25T18:25:01.781 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:01.780+0000 7f9292ffd640 1 -- 172.21.15.97:0/3764308665 <== mon.2 v1:172.21.15.97:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 82877885 0 0) 0x7f92a41dadb0 con 0x7f92a40b4d60 2024-03-25T18:25:01.781 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:01.780+0000 7f9292ffd640 1 -- 172.21.15.97:0/3764308665 --> v1:172.21.15.97:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f92a413bd10 con 0x7f92a40b4d60 2024-03-25T18:25:01.782 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:01.780+0000 7f9292ffd640 1 -- 172.21.15.97:0/3764308665 <== mon.2 v1:172.21.15.97:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f9298001c20 con 0x7f92a40b4d60 2024-03-25T18:25:01.782 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:01.780+0000 7f9292ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:01.782 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:01.780+0000 7f9292ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:25:01.782 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:01.780+0000 7f9292ffd640 10 monclient(hunting): dump: 2024-03-25T18:25:01.782 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:25:01.782 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:25:01.782 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:25:01.782 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:25:01.782 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:01.782 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:25:01.782 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:25:01.782 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:25:01.782 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:25:01.783 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:25:01.783 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:01.780+0000 7f9292ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:25:01.783 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:01.781+0000 7f9292ffd640 1 -- 172.21.15.97:0/3764308665 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2650114007 0 0) 0x7f9284004eb0 con 0x7f92a40b1730 2024-03-25T18:25:01.783 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:01.781+0000 7f9292ffd640 1 -- 172.21.15.97:0/3764308665 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f92a41dadb0 con 0x7f92a40b1730 2024-03-25T18:25:01.783 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:01.781+0000 7f9292ffd640 1 -- 172.21.15.97:0/3764308665 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f929c0019d0 con 0x7f92a40b1730 2024-03-25T18:25:01.784 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:01.781+0000 7f9292ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:01.784 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:01.781+0000 7f9292ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:25:01.784 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:01.781+0000 7f9292ffd640 10 monclient(hunting): dump: 2024-03-25T18:25:01.784 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:25:01.784 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:25:01.784 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:25:01.784 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:25:01.784 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:01.784 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:25:01.784 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:25:01.784 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:25:01.784 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:25:01.784 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:25:01.789 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:01.787+0000 7f9292ffd640 1 -- 172.21.15.97:0/3764308665 <== mon.1 v1:172.21.15.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2503756329 0 0) 0x7f92a41dadb0 con 0x7f92a40b1730 2024-03-25T18:25:01.789 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:01.787+0000 7f9292ffd640 1 -- 172.21.15.97:0/3764308665 >> v1:172.21.15.97:6790/0 conn(0x7f92a40b4d60 legacy=0x7f92a413a450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:01.789 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:01.787+0000 7f9292ffd640 1 -- 172.21.15.97:0/3764308665 >> v1:172.21.15.97:6789/0 conn(0x7f92a414be90 legacy=0x7f92a41db2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:01.789 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:01.787+0000 7f9292ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:25:01.789 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:01.787+0000 7f9292ffd640 1 monclient: found mon.b 2024-03-25T18:25:01.789 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:01.787+0000 7f9292ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:01.789 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:01.787+0000 7f9292ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:25:01.790 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:01.787+0000 7f9292ffd640 1 -- 172.21.15.97:0/3764308665 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92a413ab60 con 0x7f92a40b1730 2024-03-25T18:25:01.790 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:01.787+0000 7f9292ffd640 10 monclient: _finish_auth 0 2024-03-25T18:25:01.790 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:01.787+0000 7f9292ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:25:01.790 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:01.787+0000 7f9292ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:01.790 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:01.787+0000 7f92a89b6640 5 monclient: authenticate success, global_id 4624 2024-03-25T18:25:01.790 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:01.787+0000 7f92a89b6640 10 monclient: _renew_subs 2024-03-25T18:25:01.790 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:01.787+0000 7f92a89b6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:25:01.790 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:01.787+0000 7f92a89b6640 1 -- 172.21.15.97:0/3764308665 --> v1:172.21.15.98:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92a41db9e0 con 0x7f92a40b1730 2024-03-25T18:25:01.790 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:01.787+0000 7f92a89b6640 10 client.4624.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:25:01.790 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:01.787+0000 7f92a89b6640 10 monclient: _renew_subs 2024-03-25T18:25:01.790 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:01.787+0000 7f92a89b6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:25:01.790 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:01.787+0000 7f92a89b6640 1 -- 172.21.15.97:0/3764308665 --> v1:172.21.15.98:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f92a4139890 con 0x7f92a40b1730 2024-03-25T18:25:01.790 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:01.787+0000 7f92a89b6640 1 librados: init done 2024-03-25T18:25:01.791 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:01.788+0000 7f92a89b6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:01.791 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:01.788+0000 7f92a89b6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:01.791 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:01.788+0000 7f92a89b6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:25:01.791 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:01.788+0000 7f92a89b6640 1 -- 172.21.15.97:0/3764308665 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92a413acf0 con 0x7f92a40b1730 2024-03-25T18:25:02.056 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:02.054+0000 7f9292ffd640 1 -- 172.21.15.97:0/3764308665 <== mon.1 v1:172.21.15.98:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f929c003610 con 0x7f92a40b1730 2024-03-25T18:25:02.056 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:02.054+0000 7f9292ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:02.056 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:02.054+0000 7f9292ffd640 1 -- 172.21.15.97:0/3764308665 <== mon.1 v1:172.21.15.98:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f929c0039e0 con 0x7f92a40b1730 2024-03-25T18:25:02.056 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:02.054+0000 7f9292ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:02.057 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:02.054+0000 7f9292ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:25:02.057 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:02.054+0000 7f92a0d99640 4 set_mon_vals no callback set 2024-03-25T18:25:02.057 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:02.054+0000 7f9292ffd640 10 monclient: dump: 2024-03-25T18:25:02.057 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:25:02.057 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:25:02.057 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:25:02.057 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:25:02.057 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:02.057 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:25:02.057 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:25:02.057 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:25:02.057 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:25:02.057 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:25:02.058 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:02.055+0000 7f9292ffd640 1 -- 172.21.15.97:0/3764308665 <== mon.1 v1:172.21.15.98:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2377622137 0 0) 0x7f929c003c00 con 0x7f92a40b1730 2024-03-25T18:25:02.058 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:02.056+0000 7f92a1d9b640 10 monclient: get_auth_request con 0x7f928406dfb0 auth_method 0 2024-03-25T18:25:02.058 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:02.056+0000 7f9292ffd640 10 client.4624.objecter ms_handle_connect 0x7f928406dfb0 2024-03-25T18:25:02.063 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:02.061+0000 7f9292ffd640 1 -- 172.21.15.97:0/3764308665 <== mon.1 v1:172.21.15.98:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 430711918 0 0) 0x7f929c0547e0 con 0x7f92a40b1730 2024-03-25T18:25:02.063 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:02.061+0000 7f9292ffd640 10 client.4624.objecter ms_dispatch 0x7f92a4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:25:02.063 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:02.061+0000 7f9292ffd640 3 client.4624.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:25:02.063 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:02.061+0000 7f9292ffd640 3 client.4624.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:25:02.063 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:02.061+0000 7f9292ffd640 20 client.4624.objecter dump_active .. 0 homeless 2024-03-25T18:25:02.063 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:02.062+0000 7f9292ffd640 1 -- 172.21.15.97:0/3764308665 <== mon.1 v1:172.21.15.98:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f929c0502f0 con 0x7f92a40b1730 2024-03-25T18:25:02.063 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:02.062+0000 7f9292ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:02.063 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:02.062+0000 7f9292ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:25:02.327 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:02.325+0000 7f92a89b6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:02.327 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:02.325+0000 7f92a89b6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:02.327 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:02.325+0000 7f92a89b6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:25:02.327 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:02.325+0000 7f92a89b6640 1 -- 172.21.15.97:0/3764308665 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f92a40627b0 con 0x7f92a40b1730 2024-03-25T18:25:02.344 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:02.342+0000 7f9292ffd640 1 -- 172.21.15.97:0/3764308665 <== mon.1 v1:172.21.15.98:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2169427412) 0x7f929c052220 con 0x7f92a40b1730 2024-03-25T18:25:02.344 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:02.342+0000 7f9292ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:02.344 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:02.342+0000 7f9292ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:25:02.344 INFO:teuthology.misc.health.smithi097.stdout: 2024-03-25T18:25:02.344 INFO:teuthology.misc.health.smithi097.stdout:{"epoch":10,"fsid":"4c3d472c-79b8-486b-9e3d-2738d4306754","created":"2024-03-25T18:16:14.737309+0000","modified":"2024-03-25T18:16:20.357492+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:20.357492+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cb5152b9-262f-45c3-94b8-18d0e81df558","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6daf8793-0838-473d-a7d6-9a0d2894c75d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"136f1493-3b9c-4f43-a2e7-641482d69dae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"397d538c-4f64-4daf-9e8b-662d09f91327","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4f1df08-663f-4bda-93f8-3878ab7c24b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c4b18d70-a3ce-4b15-9b70-eae5525bd761","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8dd4033a-de0e-4f8a-96dc-59f3c1f825b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0077722e-ffa6-4a3d-9891-929d9869b286","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:25:02.347 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:02.345+0000 7f92a89b6640 10 librados: watch_flush enter 2024-03-25T18:25:02.347 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:02.345+0000 7f92a89b6640 10 librados: watch_flush exit 2024-03-25T18:25:02.347 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:02.345+0000 7f92a89b6640 20 client.4624.objecter shutdown clearing up homeless session... 2024-03-25T18:25:02.347 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:02.345+0000 7f92a89b6640 10 client.4624.objecter successfully canceled tick 2024-03-25T18:25:02.347 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:02.345+0000 7f92a89b6640 1 -- 172.21.15.97:0/3764308665 >> v1:172.21.15.98:6800/800387573 conn(0x7f928406dfb0 legacy=0x7f9284070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:02.347 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:02.345+0000 7f92a89b6640 10 monclient: shutdown 2024-03-25T18:25:02.347 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:02.345+0000 7f92a89b6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:02.347 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:02.345+0000 7f92a89b6640 1 -- 172.21.15.97:0/3764308665 >> v1:172.21.15.98:6789/0 conn(0x7f92a40b1730 legacy=0x7f92a4138240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:02.347 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:02.345+0000 7f92a89b6640 1 -- 172.21.15.97:0/3764308665 shutdown_connections 2024-03-25T18:25:02.347 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:02.345+0000 7f92a89b6640 1 -- 172.21.15.97:0/3764308665 >> 172.21.15.97:0/3764308665 conn(0x7f92a40617a0 msgr2=0x7f92a413c660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:02.348 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:02.346+0000 7f92a89b6640 1 -- 172.21.15.97:0/3764308665 shutdown_connections 2024-03-25T18:25:02.348 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:02.346+0000 7f92a89b6640 1 -- 172.21.15.97:0/3764308665 wait complete. 2024-03-25T18:25:02.348 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:02.346+0000 7f92a89b6640 1 librados: shutdown 2024-03-25T18:25:02.359 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:25:09.166 DEBUG:teuthology.orchestra.run.smithi097:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:25:09.266 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.265+0000 7fac3cd54640 10 monclient: get_monmap_and_config 2024-03-25T18:25:09.266 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.265+0000 7fac3cd54640 10 monclient: build_initial_monmap 2024-03-25T18:25:09.266 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.265+0000 7fac3cd54640 10 monclient: monmap: 2024-03-25T18:25:09.266 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:25:09.266 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:25:09.266 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:25:09.265846+0000 2024-03-25T18:25:09.266 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:25:09.265846+0000 2024-03-25T18:25:09.266 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:09.266 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:25:09.267 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:25:09.267 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T18:25:09.267 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-b 2024-03-25T18:25:09.267 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:25:09.268 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.266+0000 7fac3cd54640 1 Processor -- start 2024-03-25T18:25:09.268 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.266+0000 7fac3cd54640 1 -- start start 2024-03-25T18:25:09.268 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.266+0000 7fac3cd54640 10 monclient: init 2024-03-25T18:25:09.268 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.266+0000 7fac3cd54640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:09.268 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.266+0000 7fac3cd54640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:25:09.268 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.266+0000 7fac3cd54640 10 monclient(hunting): picked mon.noname-a con 0x7fac380b4d60 addr v1:172.21.15.97:6789/0 2024-03-25T18:25:09.268 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.266+0000 7fac3cd54640 10 monclient(hunting): picked mon.noname-c con 0x7fac380b1730 addr v1:172.21.15.98:6789/0 2024-03-25T18:25:09.268 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.267+0000 7fac3cd54640 10 monclient(hunting): picked mon.noname-b con 0x7fac3814be90 addr v1:172.21.15.97:6790/0 2024-03-25T18:25:09.268 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.267+0000 7fac3cd54640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac380af940 con 0x7fac380b4d60 2024-03-25T18:25:09.269 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.267+0000 7fac3cd54640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac380b0a70 con 0x7fac380b1730 2024-03-25T18:25:09.269 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.267+0000 7fac3cd54640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac380b0d50 con 0x7fac3814be90 2024-03-25T18:25:09.269 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.267+0000 7fac3cd54640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:09.269 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.267+0000 7fac3cd54640 10 monclient(hunting): authenticate will time out at 8121.541925s 2024-03-25T18:25:09.282 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.282+0000 7fac3659c640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fac380b4d60 0x7fac3813f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:50954/0 (socket says 172.21.15.97:50954) 2024-03-25T18:25:09.282 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.282+0000 7fac3659c640 1 -- 172.21.15.97:0/1932468914 learned_addr learned my addr 172.21.15.97:0/1932468914 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:25:09.290 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.289+0000 7fac35d9b640 1 -- 172.21.15.97:0/1932468914 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3624082453 0 0) 0x7fac380af940 con 0x7fac380b4d60 2024-03-25T18:25:09.290 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.289+0000 7fac35d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:25:09.290 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.289+0000 7fac35d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:09.290 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.289+0000 7fac35d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:09.290 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.289+0000 7fac35d9b640 10 monclient(hunting): my global_id is 4620 2024-03-25T18:25:09.290 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.289+0000 7fac35d9b640 1 -- 172.21.15.97:0/1932468914 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac24003520 con 0x7fac380b4d60 2024-03-25T18:25:09.306 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.305+0000 7fac35d9b640 1 -- 172.21.15.97:0/1932468914 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1683329270 0 0) 0x7fac24003520 con 0x7fac380b4d60 2024-03-25T18:25:09.306 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.305+0000 7fac35d9b640 1 -- 172.21.15.97:0/1932468914 >> v1:172.21.15.97:6790/0 conn(0x7fac3814be90 legacy=0x7fac3814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:25:09.306 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.305+0000 7fac35d9b640 1 -- 172.21.15.97:0/1932468914 >> v1:172.21.15.98:6789/0 conn(0x7fac380b1730 legacy=0x7fac380b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:25:09.306 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.305+0000 7fac35d9b640 10 monclient: _finish_hunting 0 2024-03-25T18:25:09.306 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.305+0000 7fac35d9b640 1 monclient: found mon.noname-a 2024-03-25T18:25:09.307 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.305+0000 7fac35d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:09.307 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.305+0000 7fac35d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.97:6789/0 2024-03-25T18:25:09.307 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.305+0000 7fac35d9b640 1 -- 172.21.15.97:0/1932468914 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac380b1030 con 0x7fac380b4d60 2024-03-25T18:25:09.307 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.305+0000 7fac35d9b640 10 monclient: _finish_auth 0 2024-03-25T18:25:09.307 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.305+0000 7fac35d9b640 10 monclient: _check_auth_tickets 2024-03-25T18:25:09.307 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.305+0000 7fac35d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:09.307 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.305+0000 7fac35d9b640 1 -- 172.21.15.97:0/1932468914 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7fac28001690 con 0x7fac380b4d60 2024-03-25T18:25:09.307 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.305+0000 7fac35d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:09.307 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.305+0000 7fac35d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:25:09.307 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.305+0000 7fac35d9b640 10 monclient: dump: 2024-03-25T18:25:09.307 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:25:09.307 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:25:09.307 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:25:09.307 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:25:09.307 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:09.308 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:25:09.308 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:25:09.308 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:25:09.308 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:25:09.308 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:25:09.308 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.305+0000 7fac3cd54640 5 monclient: authenticate success, global_id 4620 2024-03-25T18:25:09.308 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.305+0000 7fac3cd54640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:25:09.316 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.315+0000 7fac35d9b640 1 -- 172.21.15.97:0/1932468914 <== mon.0 v1:172.21.15.97:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fac28001860 con 0x7fac380b4d60 2024-03-25T18:25:09.316 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.315+0000 7fac35d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:09.316 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.315+0000 7fac35d9b640 1 -- 172.21.15.97:0/1932468914 <== mon.0 v1:172.21.15.97:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7fac28001cb0 con 0x7fac380b4d60 2024-03-25T18:25:09.316 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.315+0000 7fac3cd54640 10 monclient: get_monmap_and_config success 2024-03-25T18:25:09.316 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.315+0000 7fac3cd54640 4 set_mon_vals no callback set 2024-03-25T18:25:09.316 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.315+0000 7fac35d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:09.316 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.315+0000 7fac35d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:25:09.316 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.315+0000 7fac35d9b640 10 monclient: dump: 2024-03-25T18:25:09.316 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:25:09.317 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:25:09.317 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:25:09.317 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:25:09.317 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:09.317 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:25:09.317 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:25:09.317 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:25:09.317 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:25:09.317 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:25:09.318 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.316+0000 7fac3cd54640 10 monclient: shutdown 2024-03-25T18:25:09.319 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.316+0000 7fac3cd54640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:09.319 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.316+0000 7fac3cd54640 1 -- 172.21.15.97:0/1932468914 >> v1:172.21.15.97:6789/0 conn(0x7fac380b4d60 legacy=0x7fac3813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:09.319 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.316+0000 7fac3cd54640 1 -- 172.21.15.97:0/1932468914 shutdown_connections 2024-03-25T18:25:09.319 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.316+0000 7fac3cd54640 1 -- 172.21.15.97:0/1932468914 >> 172.21.15.97:0/1932468914 conn(0x7fac380617a0 msgr2=0x7fac38061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:09.319 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.316+0000 7fac3cd54640 1 -- 172.21.15.97:0/1932468914 shutdown_connections 2024-03-25T18:25:09.319 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.316+0000 7fac3cd54640 1 -- 172.21.15.97:0/1932468914 wait complete. 2024-03-25T18:25:09.319 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.317+0000 7fac3cd54640 10 monclient: build_initial_monmap 2024-03-25T18:25:09.319 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.317+0000 7fac3cd54640 10 monclient: monmap: 2024-03-25T18:25:09.319 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:25:09.319 INFO:teuthology.misc.health.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:25:09.319 INFO:teuthology.misc.health.smithi097.stderr:last_changed 0.000000 2024-03-25T18:25:09.320 INFO:teuthology.misc.health.smithi097.stderr:created 0.000000 2024-03-25T18:25:09.320 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:09.320 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 1 2024-03-25T18:25:09.320 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:25:09.320 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T18:25:09.320 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-c 2024-03-25T18:25:09.320 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:25:09.320 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.318+0000 7fac3cd54640 1 librados: starting msgr at 2024-03-25T18:25:09.320 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.318+0000 7fac3cd54640 1 librados: starting objecter 2024-03-25T18:25:09.320 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.318+0000 7fac3cd54640 1 Processor -- start 2024-03-25T18:25:09.320 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.318+0000 7fac3cd54640 1 -- start start 2024-03-25T18:25:09.320 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.318+0000 7fac3cd54640 1 librados: setting wanted keys 2024-03-25T18:25:09.320 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.318+0000 7fac3cd54640 1 librados: calling monclient init 2024-03-25T18:25:09.320 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.318+0000 7fac3cd54640 10 monclient: init 2024-03-25T18:25:09.321 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.318+0000 7fac3cd54640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:09.321 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.318+0000 7fac3cd54640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:25:09.321 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.318+0000 7fac3cd54640 10 monclient(hunting): picked mon.noname-c con 0x7fac380b1730 addr v1:172.21.15.97:6790/0 2024-03-25T18:25:09.321 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.319+0000 7fac3cd54640 10 monclient(hunting): picked mon.noname-a con 0x7fac380b4d60 addr v1:172.21.15.97:6789/0 2024-03-25T18:25:09.322 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.319+0000 7fac3cd54640 10 monclient(hunting): picked mon.noname-b con 0x7fac3814be90 addr v1:172.21.15.98:6789/0 2024-03-25T18:25:09.322 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.319+0000 7fac3cd54640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac380aa7f0 con 0x7fac380b4d60 2024-03-25T18:25:09.322 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.319+0000 7fac3cd54640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac380aaa60 con 0x7fac3814be90 2024-03-25T18:25:09.322 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.319+0000 7fac3cd54640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac38138840 con 0x7fac380b1730 2024-03-25T18:25:09.322 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.319+0000 7fac3cd54640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:09.322 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.319+0000 7fac3cd54640 10 monclient(hunting): authenticate will time out at 8121.593996s 2024-03-25T18:25:09.335 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.334+0000 7fac3659c640 1 --1- >> v1:172.21.15.97:6790/0 conn(0x7fac380b1730 0x7fac381d7b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.97:6790/0 says I am v1:172.21.15.97:37792/0 (socket says 172.21.15.97:37792) 2024-03-25T18:25:09.335 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.334+0000 7fac3659c640 1 -- 172.21.15.97:0/2412240675 learned_addr learned my addr 172.21.15.97:0/2412240675 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:25:09.337 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.337+0000 7fac2effd640 10 client.?.objecter ms_handle_connect 0x7fac380b1730 2024-03-25T18:25:09.337 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.337+0000 7fac2effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:09.341 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.341+0000 7fac2effd640 10 client.?.objecter ms_handle_connect 0x7fac3814be90 2024-03-25T18:25:09.341 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.341+0000 7fac2effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:09.341 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.341+0000 7fac2effd640 10 client.?.objecter ms_handle_connect 0x7fac380b4d60 2024-03-25T18:25:09.341 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.341+0000 7fac2effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:09.345 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.344+0000 7fac2effd640 1 -- 172.21.15.97:0/2412240675 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 954496723 0 0) 0x7fac38138840 con 0x7fac380b1730 2024-03-25T18:25:09.345 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.344+0000 7fac2effd640 10 monclient(hunting): _negotiate 2024-03-25T18:25:09.345 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.344+0000 7fac2effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:09.345 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.344+0000 7fac2effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:09.345 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.344+0000 7fac2effd640 10 monclient(hunting): my global_id is 4610 2024-03-25T18:25:09.345 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.344+0000 7fac2effd640 1 -- 172.21.15.97:0/2412240675 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac18004eb0 con 0x7fac380b1730 2024-03-25T18:25:09.360 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.360+0000 7fac2effd640 1 -- 172.21.15.97:0/2412240675 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3202107118 0 0) 0x7fac380aa7f0 con 0x7fac380b4d60 2024-03-25T18:25:09.360 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.360+0000 7fac2effd640 10 monclient(hunting): _negotiate 2024-03-25T18:25:09.360 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.360+0000 7fac2effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:09.360 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.360+0000 7fac2effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:09.360 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.360+0000 7fac2effd640 10 monclient(hunting): my global_id is 4623 2024-03-25T18:25:09.360 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.360+0000 7fac2effd640 1 -- 172.21.15.97:0/2412240675 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac38138840 con 0x7fac380b4d60 2024-03-25T18:25:09.362 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.362+0000 7fac2effd640 1 -- 172.21.15.97:0/2412240675 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1074427681 0 0) 0x7fac380aaa60 con 0x7fac3814be90 2024-03-25T18:25:09.363 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.362+0000 7fac2effd640 10 monclient(hunting): _negotiate 2024-03-25T18:25:09.363 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.362+0000 7fac2effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:09.363 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.362+0000 7fac2effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:09.363 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.362+0000 7fac2effd640 10 monclient(hunting): my global_id is 4627 2024-03-25T18:25:09.363 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.362+0000 7fac2effd640 1 -- 172.21.15.97:0/2412240675 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac380aa7f0 con 0x7fac3814be90 2024-03-25T18:25:09.366 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.365+0000 7fac2effd640 1 -- 172.21.15.97:0/2412240675 <== mon.2 v1:172.21.15.97:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3665841342 0 0) 0x7fac18004eb0 con 0x7fac380b1730 2024-03-25T18:25:09.366 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.365+0000 7fac2effd640 1 -- 172.21.15.97:0/2412240675 --> v1:172.21.15.97:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fac380aaa60 con 0x7fac380b1730 2024-03-25T18:25:09.366 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.365+0000 7fac2effd640 1 -- 172.21.15.97:0/2412240675 <== mon.2 v1:172.21.15.97:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7fac28001b70 con 0x7fac380b1730 2024-03-25T18:25:09.366 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.365+0000 7fac2effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:09.366 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.365+0000 7fac2effd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:25:09.366 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.365+0000 7fac2effd640 10 monclient(hunting): dump: 2024-03-25T18:25:09.366 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:25:09.366 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:25:09.366 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:25:09.366 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:25:09.366 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:09.366 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:25:09.366 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:25:09.367 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:25:09.367 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:25:09.367 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:25:09.367 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.365+0000 7fac2effd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:25:09.373 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.372+0000 7fac2effd640 1 -- 172.21.15.97:0/2412240675 <== mon.2 v1:172.21.15.97:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 227822333 0 0) 0x7fac380aaa60 con 0x7fac380b1730 2024-03-25T18:25:09.373 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.372+0000 7fac2effd640 1 -- 172.21.15.97:0/2412240675 >> v1:172.21.15.98:6789/0 conn(0x7fac3814be90 legacy=0x7fac38138130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:09.373 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.372+0000 7fac2effd640 1 -- 172.21.15.97:0/2412240675 >> v1:172.21.15.97:6789/0 conn(0x7fac380b4d60 legacy=0x7fac38137a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:09.373 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.372+0000 7fac2effd640 10 monclient: _finish_hunting 0 2024-03-25T18:25:09.373 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.372+0000 7fac2effd640 1 monclient: found mon.c 2024-03-25T18:25:09.373 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.372+0000 7fac2effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:09.373 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.372+0000 7fac2effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.97:6790/0 2024-03-25T18:25:09.373 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.372+0000 7fac2effd640 1 -- 172.21.15.97:0/2412240675 --> v1:172.21.15.97:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac3813b120 con 0x7fac380b1730 2024-03-25T18:25:09.373 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.372+0000 7fac2effd640 10 monclient: _finish_auth 0 2024-03-25T18:25:09.373 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.372+0000 7fac2effd640 10 monclient: _check_auth_tickets 2024-03-25T18:25:09.374 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.372+0000 7fac2effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:09.374 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.372+0000 7fac3cd54640 5 monclient: authenticate success, global_id 4610 2024-03-25T18:25:09.374 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.372+0000 7fac3cd54640 10 monclient: _renew_subs 2024-03-25T18:25:09.374 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.372+0000 7fac3cd54640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.97:6790/0 2024-03-25T18:25:09.374 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.372+0000 7fac3cd54640 1 -- 172.21.15.97:0/2412240675 --> v1:172.21.15.97:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac38138cd0 con 0x7fac380b1730 2024-03-25T18:25:09.374 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.372+0000 7fac3cd54640 10 client.4610.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:25:09.374 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.372+0000 7fac3cd54640 10 monclient: _renew_subs 2024-03-25T18:25:09.374 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.372+0000 7fac3cd54640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.97:6790/0 2024-03-25T18:25:09.374 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.372+0000 7fac3cd54640 1 -- 172.21.15.97:0/2412240675 --> v1:172.21.15.97:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fac381d8400 con 0x7fac380b1730 2024-03-25T18:25:09.374 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.372+0000 7fac3cd54640 1 librados: init done 2024-03-25T18:25:09.375 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.373+0000 7fac3cd54640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:09.375 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.373+0000 7fac3cd54640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:09.375 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.373+0000 7fac3cd54640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.97:6790/0 2024-03-25T18:25:09.375 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.373+0000 7fac3cd54640 1 -- 172.21.15.97:0/2412240675 --> v1:172.21.15.97:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac380b22f0 con 0x7fac380b1730 2024-03-25T18:25:09.594 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.592+0000 7fac2effd640 1 -- 172.21.15.97:0/2412240675 <== mon.2 v1:172.21.15.97:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fac28004bf0 con 0x7fac380b1730 2024-03-25T18:25:09.594 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.592+0000 7fac2effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:09.594 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.592+0000 7fac2effd640 1 -- 172.21.15.97:0/2412240675 <== mon.2 v1:172.21.15.97:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7fac28004d60 con 0x7fac380b1730 2024-03-25T18:25:09.594 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.592+0000 7fac2effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:09.594 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.592+0000 7fac2effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:25:09.594 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.592+0000 7fac2effd640 10 monclient: dump: 2024-03-25T18:25:09.594 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:25:09.594 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:25:09.595 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:25:09.595 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:25:09.595 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:09.595 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:25:09.595 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:25:09.595 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:25:09.595 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:25:09.595 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:25:09.595 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.592+0000 7fac3559a640 4 set_mon_vals no callback set 2024-03-25T18:25:09.667 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.666+0000 7fac2effd640 1 -- 172.21.15.97:0/2412240675 <== mon.2 v1:172.21.15.97:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2377622137 0 0) 0x7fac2801ad30 con 0x7fac380b1730 2024-03-25T18:25:09.667 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.667+0000 7fac2ffff640 10 monclient: get_auth_request con 0x7fac1806dda0 auth_method 0 2024-03-25T18:25:09.667 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.667+0000 7fac2effd640 10 client.4610.objecter ms_handle_connect 0x7fac1806dda0 2024-03-25T18:25:09.673 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.673+0000 7fac2effd640 1 -- 172.21.15.97:0/2412240675 <== mon.2 v1:172.21.15.97:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 430711918 0 0) 0x7fac28054810 con 0x7fac380b1730 2024-03-25T18:25:09.674 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.673+0000 7fac2effd640 10 client.4610.objecter ms_dispatch 0x7fac38000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:25:09.674 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.673+0000 7fac2effd640 3 client.4610.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:25:09.674 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.673+0000 7fac2effd640 3 client.4610.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:25:09.674 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.673+0000 7fac2effd640 20 client.4610.objecter dump_active .. 0 homeless 2024-03-25T18:25:09.674 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.673+0000 7fac2effd640 1 -- 172.21.15.97:0/2412240675 <== mon.2 v1:172.21.15.97:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fac28050320 con 0x7fac380b1730 2024-03-25T18:25:09.674 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.673+0000 7fac2effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:09.674 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.673+0000 7fac2effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:25:09.934 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.933+0000 7fac3cd54640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:09.934 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.933+0000 7fac3cd54640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:09.934 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.933+0000 7fac3cd54640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.97:6790/0 2024-03-25T18:25:09.934 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.933+0000 7fac3cd54640 1 -- 172.21.15.97:0/2412240675 --> v1:172.21.15.97:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fac380627b0 con 0x7fac380b1730 2024-03-25T18:25:09.956 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.956+0000 7fac2effd640 1 -- 172.21.15.97:0/2412240675 <== mon.2 v1:172.21.15.97:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2169427412) 0x7fac28052250 con 0x7fac380b1730 2024-03-25T18:25:09.957 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.956+0000 7fac2effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:09.957 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.956+0000 7fac2effd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:25:09.957 INFO:teuthology.misc.health.smithi097.stdout: 2024-03-25T18:25:09.957 INFO:teuthology.misc.health.smithi097.stdout:{"epoch":10,"fsid":"4c3d472c-79b8-486b-9e3d-2738d4306754","created":"2024-03-25T18:16:14.737309+0000","modified":"2024-03-25T18:16:20.357492+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:20.357492+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cb5152b9-262f-45c3-94b8-18d0e81df558","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6daf8793-0838-473d-a7d6-9a0d2894c75d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"136f1493-3b9c-4f43-a2e7-641482d69dae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"397d538c-4f64-4daf-9e8b-662d09f91327","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4f1df08-663f-4bda-93f8-3878ab7c24b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c4b18d70-a3ce-4b15-9b70-eae5525bd761","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8dd4033a-de0e-4f8a-96dc-59f3c1f825b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0077722e-ffa6-4a3d-9891-929d9869b286","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:25:09.960 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.959+0000 7fac3cd54640 10 librados: watch_flush enter 2024-03-25T18:25:09.960 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.959+0000 7fac3cd54640 10 librados: watch_flush exit 2024-03-25T18:25:09.960 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.959+0000 7fac3cd54640 20 client.4610.objecter shutdown clearing up homeless session... 2024-03-25T18:25:09.960 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.959+0000 7fac3cd54640 10 client.4610.objecter successfully canceled tick 2024-03-25T18:25:09.960 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.959+0000 7fac3cd54640 1 -- 172.21.15.97:0/2412240675 >> v1:172.21.15.98:6800/800387573 conn(0x7fac1806dda0 legacy=0x7fac18070260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:09.960 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.959+0000 7fac3cd54640 10 monclient: shutdown 2024-03-25T18:25:09.960 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.959+0000 7fac3cd54640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:09.960 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.959+0000 7fac3cd54640 1 -- 172.21.15.97:0/2412240675 >> v1:172.21.15.97:6790/0 conn(0x7fac380b1730 legacy=0x7fac381d7b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:09.960 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.959+0000 7fac3cd54640 1 -- 172.21.15.97:0/2412240675 shutdown_connections 2024-03-25T18:25:09.961 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.959+0000 7fac3cd54640 1 -- 172.21.15.97:0/2412240675 >> 172.21.15.97:0/2412240675 conn(0x7fac380617a0 msgr2=0x7fac381451c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:09.961 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.959+0000 7fac3cd54640 1 -- 172.21.15.97:0/2412240675 shutdown_connections 2024-03-25T18:25:09.961 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.959+0000 7fac3cd54640 1 -- 172.21.15.97:0/2412240675 wait complete. 2024-03-25T18:25:09.961 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:09.959+0000 7fac3cd54640 1 librados: shutdown 2024-03-25T18:25:09.971 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:25:16.778 DEBUG:teuthology.orchestra.run.smithi097:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:25:16.895 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:16.894+0000 7efd0a625640 10 monclient: get_monmap_and_config 2024-03-25T18:25:16.895 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:16.894+0000 7efd0a625640 10 monclient: build_initial_monmap 2024-03-25T18:25:16.895 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:16.894+0000 7efd0a625640 10 monclient: monmap: 2024-03-25T18:25:16.895 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:25:16.895 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:25:16.895 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:25:16.894795+0000 2024-03-25T18:25:16.895 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:25:16.894795+0000 2024-03-25T18:25:16.895 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:16.896 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:25:16.896 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:25:16.896 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T18:25:16.896 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-b 2024-03-25T18:25:16.896 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:25:16.897 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:16.895+0000 7efd0a625640 1 Processor -- start 2024-03-25T18:25:16.897 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:16.895+0000 7efd0a625640 1 -- start start 2024-03-25T18:25:16.897 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:16.895+0000 7efd0a625640 10 monclient: init 2024-03-25T18:25:16.897 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:16.896+0000 7efd0a625640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:16.897 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:16.896+0000 7efd0a625640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:25:16.897 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:16.896+0000 7efd0a625640 10 monclient(hunting): picked mon.noname-a con 0x7efd040b1ef0 addr v1:172.21.15.97:6789/0 2024-03-25T18:25:16.897 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:16.896+0000 7efd0a625640 10 monclient(hunting): picked mon.noname-b con 0x7efd041473a0 addr v1:172.21.15.97:6790/0 2024-03-25T18:25:16.897 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:16.896+0000 7efd0a625640 10 monclient(hunting): picked mon.noname-c con 0x7efd04150070 addr v1:172.21.15.98:6789/0 2024-03-25T18:25:16.897 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:16.896+0000 7efd0a625640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd040b2570 con 0x7efd040b1ef0 2024-03-25T18:25:16.897 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:16.896+0000 7efd0a625640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd04146c20 con 0x7efd04150070 2024-03-25T18:25:16.897 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:16.896+0000 7efd0a625640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd04146e80 con 0x7efd041473a0 2024-03-25T18:25:16.897 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:16.896+0000 7efd0a625640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:16.897 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:16.896+0000 7efd0a625640 10 monclient(hunting): authenticate will time out at 8129.171412s 2024-03-25T18:25:16.905 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:16.904+0000 7efd037fe640 1 --1- >> v1:172.21.15.97:6790/0 conn(0x7efd041473a0 0x7efd04143b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6790/0 says I am v1:172.21.15.97:40824/0 (socket says 172.21.15.97:40824) 2024-03-25T18:25:16.905 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:16.904+0000 7efd037fe640 1 -- 172.21.15.97:0/2847668908 learned_addr learned my addr 172.21.15.97:0/2847668908 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:25:16.915 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:16.915+0000 7efd02ffd640 1 -- 172.21.15.97:0/2847668908 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3133321228 0 0) 0x7efd04146c20 con 0x7efd04150070 2024-03-25T18:25:16.916 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:16.915+0000 7efd02ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:25:16.916 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:16.915+0000 7efd02ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:16.916 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:16.915+0000 7efd02ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:16.916 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:16.915+0000 7efd02ffd640 10 monclient(hunting): my global_id is 4630 2024-03-25T18:25:16.916 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:16.915+0000 7efd02ffd640 1 -- 172.21.15.97:0/2847668908 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efcec003520 con 0x7efd04150070 2024-03-25T18:25:16.918 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:16.917+0000 7efd02ffd640 1 -- 172.21.15.97:0/2847668908 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1656999907 0 0) 0x7efd04146e80 con 0x7efd041473a0 2024-03-25T18:25:16.918 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:16.918+0000 7efd02ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:25:16.918 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:16.918+0000 7efd02ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:16.918 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:16.918+0000 7efd02ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:16.918 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:16.918+0000 7efd02ffd640 10 monclient(hunting): my global_id is 4613 2024-03-25T18:25:16.918 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:16.918+0000 7efd02ffd640 1 -- 172.21.15.97:0/2847668908 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd04146c20 con 0x7efd041473a0 2024-03-25T18:25:16.922 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:16.921+0000 7efd02ffd640 1 -- 172.21.15.97:0/2847668908 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3266026176 0 0) 0x7efcec003520 con 0x7efd04150070 2024-03-25T18:25:16.922 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:16.921+0000 7efd02ffd640 1 -- 172.21.15.97:0/2847668908 >> v1:172.21.15.97:6790/0 conn(0x7efd041473a0 legacy=0x7efd04143b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:16.922 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:16.921+0000 7efd02ffd640 1 -- 172.21.15.97:0/2847668908 >> v1:172.21.15.97:6789/0 conn(0x7efd040b1ef0 legacy=0x7efd040b28a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:16.922 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:16.921+0000 7efd02ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:25:16.922 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:16.921+0000 7efd02ffd640 1 monclient: found mon.noname-c 2024-03-25T18:25:16.922 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:16.921+0000 7efd02ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:16.922 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:16.921+0000 7efd02ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.98:6789/0 2024-03-25T18:25:16.922 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:16.921+0000 7efd02ffd640 1 -- 172.21.15.97:0/2847668908 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd0406e1b0 con 0x7efd04150070 2024-03-25T18:25:16.923 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:16.921+0000 7efd02ffd640 10 monclient: _finish_auth 0 2024-03-25T18:25:16.923 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:16.921+0000 7efd02ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:25:16.923 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:16.921+0000 7efd02ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:16.923 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:16.921+0000 7efd02ffd640 1 -- 172.21.15.97:0/2847668908 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7efcfc001a10 con 0x7efd04150070 2024-03-25T18:25:16.923 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:16.921+0000 7efd0a625640 5 monclient: authenticate success, global_id 4630 2024-03-25T18:25:16.923 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:16.921+0000 7efd0a625640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:25:16.923 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:16.921+0000 7efd02ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:16.923 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:16.921+0000 7efd02ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:25:16.923 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:16.921+0000 7efd02ffd640 10 monclient: dump: 2024-03-25T18:25:16.923 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:25:16.923 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:25:16.923 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:25:16.923 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:25:16.923 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:16.924 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:25:16.924 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:25:16.924 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:25:16.924 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:25:16.924 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:25:16.924 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:16.921+0000 7efd0a625640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:25:16.931 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:16.931+0000 7efd02ffd640 1 -- 172.21.15.97:0/2847668908 <== mon.1 v1:172.21.15.98:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efcfc001b70 con 0x7efd04150070 2024-03-25T18:25:16.931 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:16.931+0000 7efd02ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:16.931 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:16.931+0000 7efd02ffd640 1 -- 172.21.15.97:0/2847668908 <== mon.1 v1:172.21.15.98:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7efcfc001f80 con 0x7efd04150070 2024-03-25T18:25:16.931 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:16.931+0000 7efd02ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:16.932 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:16.931+0000 7efd02ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:25:16.932 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:16.931+0000 7efd02ffd640 10 monclient: dump: 2024-03-25T18:25:16.932 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:25:16.932 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:25:16.932 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:25:16.932 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:25:16.932 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:16.932 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:25:16.932 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:25:16.932 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:25:16.932 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:25:16.932 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:25:16.932 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:16.931+0000 7efd0a625640 10 monclient: get_monmap_and_config success 2024-03-25T18:25:16.932 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:16.931+0000 7efd0a625640 4 set_mon_vals no callback set 2024-03-25T18:25:16.934 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:16.932+0000 7efd0a625640 10 monclient: shutdown 2024-03-25T18:25:16.934 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:16.932+0000 7efd0a625640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:16.934 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:16.932+0000 7efd0a625640 1 -- 172.21.15.97:0/2847668908 >> v1:172.21.15.98:6789/0 conn(0x7efd04150070 legacy=0x7efd04152460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:16.934 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:16.932+0000 7efd0a625640 1 -- 172.21.15.97:0/2847668908 shutdown_connections 2024-03-25T18:25:16.934 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:16.932+0000 7efd0a625640 1 -- 172.21.15.97:0/2847668908 >> 172.21.15.97:0/2847668908 conn(0x7efd040b7ea0 msgr2=0x7efd040b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:16.934 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:16.932+0000 7efd0a625640 1 -- 172.21.15.97:0/2847668908 shutdown_connections 2024-03-25T18:25:16.934 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:16.932+0000 7efd0a625640 1 -- 172.21.15.97:0/2847668908 wait complete. 2024-03-25T18:25:16.934 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:16.933+0000 7efd0a625640 10 monclient: build_initial_monmap 2024-03-25T18:25:16.935 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:16.933+0000 7efd0a625640 10 monclient: monmap: 2024-03-25T18:25:16.935 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:25:16.935 INFO:teuthology.misc.health.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:25:16.935 INFO:teuthology.misc.health.smithi097.stderr:last_changed 0.000000 2024-03-25T18:25:16.935 INFO:teuthology.misc.health.smithi097.stderr:created 0.000000 2024-03-25T18:25:16.935 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:16.935 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 1 2024-03-25T18:25:16.935 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:25:16.935 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T18:25:16.935 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-c 2024-03-25T18:25:16.935 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:25:16.935 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:16.933+0000 7efd0a625640 1 librados: starting msgr at 2024-03-25T18:25:16.935 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:16.933+0000 7efd0a625640 1 librados: starting objecter 2024-03-25T18:25:16.935 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:16.933+0000 7efd0a625640 1 Processor -- start 2024-03-25T18:25:16.935 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:16.933+0000 7efd0a625640 1 -- start start 2024-03-25T18:25:16.936 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:16.933+0000 7efd0a625640 1 librados: setting wanted keys 2024-03-25T18:25:16.936 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:16.933+0000 7efd0a625640 1 librados: calling monclient init 2024-03-25T18:25:16.936 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:16.933+0000 7efd0a625640 10 monclient: init 2024-03-25T18:25:16.937 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:16.934+0000 7efd0a625640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:16.937 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:16.934+0000 7efd0a625640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:25:16.937 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:16.934+0000 7efd0a625640 10 monclient(hunting): picked mon.noname-a con 0x7efd040b1ef0 addr v1:172.21.15.97:6789/0 2024-03-25T18:25:16.937 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:16.934+0000 7efd0a625640 10 monclient(hunting): picked mon.noname-c con 0x7efd041473a0 addr v1:172.21.15.97:6790/0 2024-03-25T18:25:16.937 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:16.934+0000 7efd0a625640 10 monclient(hunting): picked mon.noname-b con 0x7efd04150070 addr v1:172.21.15.98:6789/0 2024-03-25T18:25:16.937 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:16.934+0000 7efd0a625640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd0413ce70 con 0x7efd040b1ef0 2024-03-25T18:25:16.937 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:16.934+0000 7efd0a625640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd0413cbf0 con 0x7efd04150070 2024-03-25T18:25:16.937 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:16.934+0000 7efd0a625640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd04149990 con 0x7efd041473a0 2024-03-25T18:25:16.937 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:16.934+0000 7efd0a625640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:16.937 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:16.934+0000 7efd0a625640 10 monclient(hunting): authenticate will time out at 8129.209488s 2024-03-25T18:25:16.949 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:16.949+0000 7efd08bc2640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7efd04150070 0x7efd0413c4e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.97:59712/0 (socket says 172.21.15.97:59712) 2024-03-25T18:25:16.949 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:16.949+0000 7efd08bc2640 1 -- 172.21.15.97:0/2529875791 learned_addr learned my addr 172.21.15.97:0/2529875791 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:25:16.950 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:16.949+0000 7efd00ff9640 10 client.?.objecter ms_handle_connect 0x7efd04150070 2024-03-25T18:25:16.950 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:16.950+0000 7efd00ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:16.954 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:16.953+0000 7efd00ff9640 10 client.?.objecter ms_handle_connect 0x7efd040b1ef0 2024-03-25T18:25:16.954 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:16.953+0000 7efd00ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:16.956 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:16.955+0000 7efd00ff9640 10 client.?.objecter ms_handle_connect 0x7efd041473a0 2024-03-25T18:25:16.956 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:16.955+0000 7efd00ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:16.962 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:16.961+0000 7efd00ff9640 1 -- 172.21.15.97:0/2529875791 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3483877003 0 0) 0x7efd0413ce70 con 0x7efd040b1ef0 2024-03-25T18:25:16.962 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:16.961+0000 7efd00ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:25:16.962 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:16.961+0000 7efd00ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:16.962 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:16.961+0000 7efd00ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:16.962 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:16.961+0000 7efd00ff9640 10 monclient(hunting): my global_id is 4629 2024-03-25T18:25:16.962 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:16.961+0000 7efd00ff9640 1 -- 172.21.15.97:0/2529875791 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efcf0004eb0 con 0x7efd040b1ef0 2024-03-25T18:25:16.963 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:16.962+0000 7efd00ff9640 1 -- 172.21.15.97:0/2529875791 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 841152557 0 0) 0x7efd0413cbf0 con 0x7efd04150070 2024-03-25T18:25:16.963 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:16.962+0000 7efd00ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:25:16.963 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:16.962+0000 7efd00ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:16.963 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:16.962+0000 7efd00ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:16.963 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:16.962+0000 7efd00ff9640 10 monclient(hunting): my global_id is 4633 2024-03-25T18:25:16.963 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:16.962+0000 7efd00ff9640 1 -- 172.21.15.97:0/2529875791 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd0413ce70 con 0x7efd04150070 2024-03-25T18:25:16.969 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:16.969+0000 7efd00ff9640 1 -- 172.21.15.97:0/2529875791 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3478423711 0 0) 0x7efcf0004eb0 con 0x7efd040b1ef0 2024-03-25T18:25:16.969 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:16.969+0000 7efd00ff9640 1 -- 172.21.15.97:0/2529875791 --> v1:172.21.15.97:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efd0413cbf0 con 0x7efd040b1ef0 2024-03-25T18:25:16.975 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:16.975+0000 7efd00ff9640 1 -- 172.21.15.97:0/2529875791 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 332065271 0 0) 0x7efd04149990 con 0x7efd041473a0 2024-03-25T18:25:16.975 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:16.975+0000 7efd00ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:25:16.975 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:16.975+0000 7efd00ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:16.976 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:16.975+0000 7efd00ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:16.976 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:16.975+0000 7efd00ff9640 10 monclient(hunting): my global_id is 4616 2024-03-25T18:25:16.976 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:16.975+0000 7efd00ff9640 1 -- 172.21.15.97:0/2529875791 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efcf0004eb0 con 0x7efd041473a0 2024-03-25T18:25:16.976 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:16.975+0000 7efd00ff9640 1 -- 172.21.15.97:0/2529875791 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7efcf40019d0 con 0x7efd040b1ef0 2024-03-25T18:25:16.976 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:16.975+0000 7efd00ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:16.976 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:16.975+0000 7efd00ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:25:16.976 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:16.975+0000 7efd00ff9640 10 monclient(hunting): dump: 2024-03-25T18:25:16.976 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:25:16.976 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:25:16.976 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:25:16.977 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:25:16.977 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:16.977 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:25:16.977 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:25:16.977 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:25:16.977 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:25:16.977 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:25:16.977 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:16.975+0000 7efd00ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:25:16.984 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:16.983+0000 7efd00ff9640 1 -- 172.21.15.97:0/2529875791 <== mon.0 v1:172.21.15.97:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3697440595 0 0) 0x7efd0413cbf0 con 0x7efd040b1ef0 2024-03-25T18:25:16.984 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:16.983+0000 7efd00ff9640 1 -- 172.21.15.97:0/2529875791 >> v1:172.21.15.97:6790/0 conn(0x7efd041473a0 legacy=0x7efd0413bdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:16.984 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:16.983+0000 7efd00ff9640 1 -- 172.21.15.97:0/2529875791 >> v1:172.21.15.98:6789/0 conn(0x7efd04150070 legacy=0x7efd0413c4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:16.984 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:16.983+0000 7efd00ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:25:16.984 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:16.983+0000 7efd00ff9640 1 monclient: found mon.a 2024-03-25T18:25:16.985 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:16.983+0000 7efd00ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:16.985 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:16.983+0000 7efd00ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:25:16.985 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:16.983+0000 7efd00ff9640 1 -- 172.21.15.97:0/2529875791 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd0414b580 con 0x7efd040b1ef0 2024-03-25T18:25:16.985 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:16.983+0000 7efd00ff9640 10 monclient: _finish_auth 0 2024-03-25T18:25:16.985 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:16.983+0000 7efd00ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:25:16.985 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:16.983+0000 7efd00ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:16.985 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:16.983+0000 7efd0a625640 5 monclient: authenticate success, global_id 4629 2024-03-25T18:25:16.985 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:16.983+0000 7efd0a625640 10 monclient: _renew_subs 2024-03-25T18:25:16.985 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:16.983+0000 7efd0a625640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:25:16.985 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:16.983+0000 7efd0a625640 1 -- 172.21.15.97:0/2529875791 --> v1:172.21.15.97:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd0414b860 con 0x7efd040b1ef0 2024-03-25T18:25:16.985 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:16.983+0000 7efd0a625640 10 client.4629.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:25:16.985 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:16.983+0000 7efd0a625640 10 monclient: _renew_subs 2024-03-25T18:25:16.985 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:16.983+0000 7efd0a625640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:25:16.985 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:16.983+0000 7efd0a625640 1 -- 172.21.15.97:0/2529875791 --> v1:172.21.15.97:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efd04149470 con 0x7efd040b1ef0 2024-03-25T18:25:16.985 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:16.983+0000 7efd0a625640 1 librados: init done 2024-03-25T18:25:16.986 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:16.985+0000 7efd0a625640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:16.986 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:16.985+0000 7efd0a625640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:16.986 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:16.985+0000 7efd0a625640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:25:16.986 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:16.985+0000 7efd0a625640 1 -- 172.21.15.97:0/2529875791 --> v1:172.21.15.97:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd040627b0 con 0x7efd040b1ef0 2024-03-25T18:25:17.289 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:17.288+0000 7efd00ff9640 1 -- 172.21.15.97:0/2529875791 <== mon.0 v1:172.21.15.97:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efcf4003610 con 0x7efd040b1ef0 2024-03-25T18:25:17.289 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:17.288+0000 7efd00ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:17.289 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:17.288+0000 7efd00ff9640 1 -- 172.21.15.97:0/2529875791 <== mon.0 v1:172.21.15.97:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7efcf4003a60 con 0x7efd040b1ef0 2024-03-25T18:25:17.289 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:17.288+0000 7efd00ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:17.289 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:17.288+0000 7efd00ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:25:17.289 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:17.288+0000 7efd00ff9640 10 monclient: dump: 2024-03-25T18:25:17.289 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:25:17.289 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:25:17.290 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:25:17.290 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:25:17.290 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:17.290 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:25:17.290 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:25:17.290 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:25:17.290 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:25:17.290 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:25:17.290 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:17.288+0000 7efd027fc640 4 set_mon_vals no callback set 2024-03-25T18:25:17.291 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:17.289+0000 7efd00ff9640 1 -- 172.21.15.97:0/2529875791 <== mon.0 v1:172.21.15.97:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2377622137 0 0) 0x7efcf4003c80 con 0x7efd040b1ef0 2024-03-25T18:25:17.291 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:17.290+0000 7efd037fe640 10 monclient: get_auth_request con 0x7efcf006de40 auth_method 0 2024-03-25T18:25:17.291 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:17.290+0000 7efd00ff9640 10 client.4629.objecter ms_handle_connect 0x7efcf006de40 2024-03-25T18:25:17.297 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:17.296+0000 7efd00ff9640 1 -- 172.21.15.97:0/2529875791 <== mon.0 v1:172.21.15.97:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 430711918 0 0) 0x7efcf40535e0 con 0x7efd040b1ef0 2024-03-25T18:25:17.297 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:17.296+0000 7efd00ff9640 10 client.4629.objecter ms_dispatch 0x7efd04000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:25:17.297 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:17.296+0000 7efd00ff9640 3 client.4629.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:25:17.297 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:17.296+0000 7efd00ff9640 3 client.4629.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:25:17.297 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:17.296+0000 7efd00ff9640 20 client.4629.objecter dump_active .. 0 homeless 2024-03-25T18:25:17.297 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:17.296+0000 7efd00ff9640 1 -- 172.21.15.97:0/2529875791 <== mon.0 v1:172.21.15.97:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7efcf404f0f0 con 0x7efd040b1ef0 2024-03-25T18:25:17.297 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:17.296+0000 7efd00ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:17.297 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:17.296+0000 7efd00ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:25:17.551 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:17.550+0000 7efd0a625640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:17.551 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:17.550+0000 7efd0a625640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:17.551 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:17.550+0000 7efd0a625640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:25:17.551 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:17.550+0000 7efd0a625640 1 -- 172.21.15.97:0/2529875791 --> v1:172.21.15.97:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efd04142320 con 0x7efd040b1ef0 2024-03-25T18:25:17.628 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:17.627+0000 7efd00ff9640 1 -- 172.21.15.97:0/2529875791 <== mon.0 v1:172.21.15.97:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2169427412) 0x7efcf4051020 con 0x7efd040b1ef0 2024-03-25T18:25:17.628 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:17.627+0000 7efd00ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:17.628 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:17.627+0000 7efd00ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:25:17.629 INFO:teuthology.misc.health.smithi097.stdout: 2024-03-25T18:25:17.629 INFO:teuthology.misc.health.smithi097.stdout:{"epoch":10,"fsid":"4c3d472c-79b8-486b-9e3d-2738d4306754","created":"2024-03-25T18:16:14.737309+0000","modified":"2024-03-25T18:16:20.357492+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:20.357492+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cb5152b9-262f-45c3-94b8-18d0e81df558","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6daf8793-0838-473d-a7d6-9a0d2894c75d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"136f1493-3b9c-4f43-a2e7-641482d69dae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"397d538c-4f64-4daf-9e8b-662d09f91327","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4f1df08-663f-4bda-93f8-3878ab7c24b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c4b18d70-a3ce-4b15-9b70-eae5525bd761","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8dd4033a-de0e-4f8a-96dc-59f3c1f825b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0077722e-ffa6-4a3d-9891-929d9869b286","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:25:17.631 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:17.630+0000 7efd0a625640 10 librados: watch_flush enter 2024-03-25T18:25:17.631 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:17.630+0000 7efd0a625640 10 librados: watch_flush exit 2024-03-25T18:25:17.631 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:17.630+0000 7efd0a625640 20 client.4629.objecter shutdown clearing up homeless session... 2024-03-25T18:25:17.631 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:17.630+0000 7efd0a625640 10 client.4629.objecter successfully canceled tick 2024-03-25T18:25:17.631 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:17.630+0000 7efd0a625640 1 -- 172.21.15.97:0/2529875791 >> v1:172.21.15.98:6800/800387573 conn(0x7efcf006de40 legacy=0x7efcf0070300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:17.631 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:17.630+0000 7efd0a625640 10 monclient: shutdown 2024-03-25T18:25:17.631 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:17.630+0000 7efd0a625640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:17.632 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:17.630+0000 7efd0a625640 1 -- 172.21.15.97:0/2529875791 >> v1:172.21.15.97:6789/0 conn(0x7efd040b1ef0 legacy=0x7efd04149bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:17.632 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:17.630+0000 7efd0a625640 1 -- 172.21.15.97:0/2529875791 shutdown_connections 2024-03-25T18:25:17.632 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:17.630+0000 7efd0a625640 1 -- 172.21.15.97:0/2529875791 >> 172.21.15.97:0/2529875791 conn(0x7efd040b7ea0 msgr2=0x7efd040b70f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:17.632 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:17.630+0000 7efd0a625640 1 -- 172.21.15.97:0/2529875791 shutdown_connections 2024-03-25T18:25:17.632 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:17.630+0000 7efd0a625640 1 -- 172.21.15.97:0/2529875791 wait complete. 2024-03-25T18:25:17.632 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:17.630+0000 7efd0a625640 1 librados: shutdown 2024-03-25T18:25:17.642 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:25:18.587 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:09:20.625 34931== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:18.591 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:09:20.630 34931== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:18.633 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:09:20.671 34931== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:18.635 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:09:20.673 34935== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:18.643 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:09:20.686 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:18.649 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:09:20.687 34935== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:18.651 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:09:20.694 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:18.671 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:09:20.714 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:18.672 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:09:20.711 34935== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:24.448 DEBUG:teuthology.orchestra.run.smithi097:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:25:24.561 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.559+0000 7fa4f8eb5640 10 monclient: get_monmap_and_config 2024-03-25T18:25:24.562 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.559+0000 7fa4f8eb5640 10 monclient: build_initial_monmap 2024-03-25T18:25:24.562 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.559+0000 7fa4f8eb5640 10 monclient: monmap: 2024-03-25T18:25:24.562 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:25:24.562 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:25:24.562 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:25:24.561163+0000 2024-03-25T18:25:24.562 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:25:24.561163+0000 2024-03-25T18:25:24.562 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:24.562 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:25:24.562 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:25:24.562 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T18:25:24.562 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-b 2024-03-25T18:25:24.562 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:25:24.563 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.560+0000 7fa4f8eb5640 1 Processor -- start 2024-03-25T18:25:24.563 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.560+0000 7fa4f8eb5640 1 -- start start 2024-03-25T18:25:24.563 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.560+0000 7fa4f8eb5640 10 monclient: init 2024-03-25T18:25:24.563 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.561+0000 7fa4f8eb5640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:24.563 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.561+0000 7fa4f8eb5640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:25:24.564 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.561+0000 7fa4f8eb5640 10 monclient(hunting): picked mon.noname-b con 0x7fa4f40a98f0 addr v1:172.21.15.97:6790/0 2024-03-25T18:25:24.564 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.561+0000 7fa4f8eb5640 10 monclient(hunting): picked mon.noname-c con 0x7fa4f413f710 addr v1:172.21.15.98:6789/0 2024-03-25T18:25:24.564 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.561+0000 7fa4f8eb5640 10 monclient(hunting): picked mon.noname-a con 0x7fa4f41430c0 addr v1:172.21.15.97:6789/0 2024-03-25T18:25:24.564 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.561+0000 7fa4f8eb5640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4f4142520 con 0x7fa4f41430c0 2024-03-25T18:25:24.564 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.561+0000 7fa4f8eb5640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4f40b28a0 con 0x7fa4f413f710 2024-03-25T18:25:24.564 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.561+0000 7fa4f8eb5640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4f40b2a30 con 0x7fa4f40a98f0 2024-03-25T18:25:24.564 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.561+0000 7fa4f8eb5640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:24.564 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.561+0000 7fa4f8eb5640 10 monclient(hunting): authenticate will time out at 8136.837810s 2024-03-25T18:25:24.576 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.575+0000 7fa4f2d9d640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fa4f41430c0 0x7fa4f414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:33858/0 (socket says 172.21.15.97:33858) 2024-03-25T18:25:24.576 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.575+0000 7fa4f2d9d640 1 -- 172.21.15.97:0/2375198716 learned_addr learned my addr 172.21.15.97:0/2375198716 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:25:24.582 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.580+0000 7fa4f159a640 1 -- 172.21.15.97:0/2375198716 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3263685292 0 0) 0x7fa4f4142520 con 0x7fa4f41430c0 2024-03-25T18:25:24.582 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.580+0000 7fa4f159a640 10 monclient(hunting): _negotiate 2024-03-25T18:25:24.582 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.580+0000 7fa4f159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:24.582 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.580+0000 7fa4f159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:24.582 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.580+0000 7fa4f159a640 10 monclient(hunting): my global_id is 4632 2024-03-25T18:25:24.582 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.580+0000 7fa4f159a640 1 -- 172.21.15.97:0/2375198716 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4dc003520 con 0x7fa4f41430c0 2024-03-25T18:25:24.588 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.586+0000 7fa4f159a640 1 -- 172.21.15.97:0/2375198716 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2294929103 0 0) 0x7fa4f40b28a0 con 0x7fa4f413f710 2024-03-25T18:25:24.588 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.586+0000 7fa4f159a640 10 monclient(hunting): _negotiate 2024-03-25T18:25:24.588 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.586+0000 7fa4f159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:24.588 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.586+0000 7fa4f159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:24.588 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.586+0000 7fa4f159a640 10 monclient(hunting): my global_id is 4636 2024-03-25T18:25:24.588 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.586+0000 7fa4f159a640 1 -- 172.21.15.97:0/2375198716 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4f4142520 con 0x7fa4f413f710 2024-03-25T18:25:24.591 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.588+0000 7fa4f159a640 1 -- 172.21.15.97:0/2375198716 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3368308114 0 0) 0x7fa4dc003520 con 0x7fa4f41430c0 2024-03-25T18:25:24.591 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.588+0000 7fa4f159a640 1 -- 172.21.15.97:0/2375198716 >> v1:172.21.15.97:6790/0 conn(0x7fa4f40a98f0 legacy=0x7fa4f40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:24.591 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.588+0000 7fa4f159a640 1 -- 172.21.15.97:0/2375198716 >> v1:172.21.15.98:6789/0 conn(0x7fa4f413f710 legacy=0x7fa4f41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:24.591 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.588+0000 7fa4f159a640 10 monclient: _finish_hunting 0 2024-03-25T18:25:24.591 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.588+0000 7fa4f159a640 1 monclient: found mon.noname-a 2024-03-25T18:25:24.591 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.588+0000 7fa4f159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:24.591 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.588+0000 7fa4f159a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.97:6789/0 2024-03-25T18:25:24.591 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.588+0000 7fa4f159a640 1 -- 172.21.15.97:0/2375198716 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4f40b44c0 con 0x7fa4f41430c0 2024-03-25T18:25:24.591 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.589+0000 7fa4f159a640 10 monclient: _finish_auth 0 2024-03-25T18:25:24.591 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.589+0000 7fa4f159a640 10 monclient: _check_auth_tickets 2024-03-25T18:25:24.591 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.589+0000 7fa4f159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:24.591 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.589+0000 7fa4f159a640 1 -- 172.21.15.97:0/2375198716 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7fa4ec001a10 con 0x7fa4f41430c0 2024-03-25T18:25:24.592 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.589+0000 7fa4f159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:24.592 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.589+0000 7fa4f159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:25:24.592 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.589+0000 7fa4f159a640 10 monclient: dump: 2024-03-25T18:25:24.592 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:25:24.592 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:25:24.592 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:25:24.592 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:25:24.592 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:24.592 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:25:24.592 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:25:24.592 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:25:24.592 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:25:24.592 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:25:24.592 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.589+0000 7fa4f8eb5640 5 monclient: authenticate success, global_id 4632 2024-03-25T18:25:24.593 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.589+0000 7fa4f8eb5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:25:24.605 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.603+0000 7fa4f159a640 1 -- 172.21.15.97:0/2375198716 <== mon.0 v1:172.21.15.97:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa4ec001b70 con 0x7fa4f41430c0 2024-03-25T18:25:24.605 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.603+0000 7fa4f159a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:24.605 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.603+0000 7fa4f159a640 1 -- 172.21.15.97:0/2375198716 <== mon.0 v1:172.21.15.97:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7fa4ec001f80 con 0x7fa4f41430c0 2024-03-25T18:25:24.605 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.603+0000 7fa4f8eb5640 10 monclient: get_monmap_and_config success 2024-03-25T18:25:24.605 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.603+0000 7fa4f8eb5640 4 set_mon_vals no callback set 2024-03-25T18:25:24.605 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.603+0000 7fa4f159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:24.605 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.603+0000 7fa4f159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:25:24.605 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.603+0000 7fa4f159a640 10 monclient: dump: 2024-03-25T18:25:24.605 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:25:24.605 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:25:24.605 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:25:24.606 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:25:24.606 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:24.606 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:25:24.606 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:25:24.606 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:25:24.606 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:25:24.606 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:25:24.609 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.604+0000 7fa4f8eb5640 10 monclient: shutdown 2024-03-25T18:25:24.609 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.604+0000 7fa4f8eb5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:24.609 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.604+0000 7fa4f8eb5640 1 -- 172.21.15.97:0/2375198716 >> v1:172.21.15.97:6789/0 conn(0x7fa4f41430c0 legacy=0x7fa4f414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:24.609 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.604+0000 7fa4f8eb5640 1 -- 172.21.15.97:0/2375198716 shutdown_connections 2024-03-25T18:25:24.609 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.604+0000 7fa4f8eb5640 1 -- 172.21.15.97:0/2375198716 >> 172.21.15.97:0/2375198716 conn(0x7fa4f40b7ea0 msgr2=0x7fa4f40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:24.609 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.604+0000 7fa4f8eb5640 1 -- 172.21.15.97:0/2375198716 shutdown_connections 2024-03-25T18:25:24.609 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.604+0000 7fa4f8eb5640 1 -- 172.21.15.97:0/2375198716 wait complete. 2024-03-25T18:25:24.610 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.605+0000 7fa4f8eb5640 10 monclient: build_initial_monmap 2024-03-25T18:25:24.610 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.605+0000 7fa4f8eb5640 10 monclient: monmap: 2024-03-25T18:25:24.610 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:25:24.610 INFO:teuthology.misc.health.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:25:24.610 INFO:teuthology.misc.health.smithi097.stderr:last_changed 0.000000 2024-03-25T18:25:24.610 INFO:teuthology.misc.health.smithi097.stderr:created 0.000000 2024-03-25T18:25:24.610 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:24.610 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 1 2024-03-25T18:25:24.610 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:25:24.610 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T18:25:24.610 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-c 2024-03-25T18:25:24.610 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:25:24.610 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.606+0000 7fa4f8eb5640 1 librados: starting msgr at 2024-03-25T18:25:24.610 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.606+0000 7fa4f8eb5640 1 librados: starting objecter 2024-03-25T18:25:24.610 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.606+0000 7fa4f8eb5640 1 Processor -- start 2024-03-25T18:25:24.611 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.606+0000 7fa4f8eb5640 1 -- start start 2024-03-25T18:25:24.611 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.606+0000 7fa4f8eb5640 1 librados: setting wanted keys 2024-03-25T18:25:24.611 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.606+0000 7fa4f8eb5640 1 librados: calling monclient init 2024-03-25T18:25:24.611 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.606+0000 7fa4f8eb5640 10 monclient: init 2024-03-25T18:25:24.611 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.607+0000 7fa4f8eb5640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:24.611 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.607+0000 7fa4f8eb5640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:25:24.611 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.607+0000 7fa4f8eb5640 10 monclient(hunting): picked mon.noname-b con 0x7fa4f40a98f0 addr v1:172.21.15.98:6789/0 2024-03-25T18:25:24.611 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.607+0000 7fa4f8eb5640 10 monclient(hunting): picked mon.noname-c con 0x7fa4f413f710 addr v1:172.21.15.97:6790/0 2024-03-25T18:25:24.611 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.607+0000 7fa4f8eb5640 10 monclient(hunting): picked mon.noname-a con 0x7fa4f41430c0 addr v1:172.21.15.97:6789/0 2024-03-25T18:25:24.611 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.607+0000 7fa4f8eb5640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4f41d7df0 con 0x7fa4f41430c0 2024-03-25T18:25:24.611 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.607+0000 7fa4f8eb5640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4f41d7f80 con 0x7fa4f40a98f0 2024-03-25T18:25:24.611 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.607+0000 7fa4f8eb5640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4f413b5c0 con 0x7fa4f413f710 2024-03-25T18:25:24.611 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.607+0000 7fa4f8eb5640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:24.611 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.607+0000 7fa4f8eb5640 10 monclient(hunting): authenticate will time out at 8136.883249s 2024-03-25T18:25:24.628 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.626+0000 7fa4f2d9d640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fa4f41430c0 0x7fa4f4135090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:33916/0 (socket says 172.21.15.97:33916) 2024-03-25T18:25:24.628 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.626+0000 7fa4f2d9d640 1 -- 172.21.15.97:0/3798363168 learned_addr learned my addr 172.21.15.97:0/3798363168 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:25:24.631 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.630+0000 7fa4e2ffd640 10 client.?.objecter ms_handle_connect 0x7fa4f41430c0 2024-03-25T18:25:24.631 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.630+0000 7fa4e2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:24.639 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.638+0000 7fa4e2ffd640 10 client.?.objecter ms_handle_connect 0x7fa4f413f710 2024-03-25T18:25:24.639 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.638+0000 7fa4e2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:24.640 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.639+0000 7fa4e2ffd640 10 client.?.objecter ms_handle_connect 0x7fa4f40a98f0 2024-03-25T18:25:24.640 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.639+0000 7fa4e2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:24.645 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.643+0000 7fa4e2ffd640 1 -- 172.21.15.97:0/3798363168 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1384290127 0 0) 0x7fa4f41d7df0 con 0x7fa4f41430c0 2024-03-25T18:25:24.645 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.643+0000 7fa4e2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:25:24.645 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.643+0000 7fa4e2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:24.645 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.643+0000 7fa4e2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:24.645 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.643+0000 7fa4e2ffd640 10 monclient(hunting): my global_id is 4635 2024-03-25T18:25:24.645 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.643+0000 7fa4e2ffd640 1 -- 172.21.15.97:0/3798363168 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4d4004eb0 con 0x7fa4f41430c0 2024-03-25T18:25:24.646 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.644+0000 7fa4e2ffd640 1 -- 172.21.15.97:0/3798363168 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4008289440 0 0) 0x7fa4f413b5c0 con 0x7fa4f413f710 2024-03-25T18:25:24.646 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.644+0000 7fa4e2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:25:24.646 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.644+0000 7fa4e2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:24.646 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.644+0000 7fa4e2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:24.646 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.644+0000 7fa4e2ffd640 10 monclient(hunting): my global_id is 4622 2024-03-25T18:25:24.646 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.644+0000 7fa4e2ffd640 1 -- 172.21.15.97:0/3798363168 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4f41d7df0 con 0x7fa4f413f710 2024-03-25T18:25:24.649 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.648+0000 7fa4e2ffd640 1 -- 172.21.15.97:0/3798363168 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 703540591 0 0) 0x7fa4f41d7f80 con 0x7fa4f40a98f0 2024-03-25T18:25:24.650 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.648+0000 7fa4e2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:25:24.650 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.648+0000 7fa4e2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:24.650 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.648+0000 7fa4e2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:24.650 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.648+0000 7fa4e2ffd640 10 monclient(hunting): my global_id is 4639 2024-03-25T18:25:24.650 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.648+0000 7fa4e2ffd640 1 -- 172.21.15.97:0/3798363168 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4f413b5c0 con 0x7fa4f40a98f0 2024-03-25T18:25:24.663 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.661+0000 7fa4e2ffd640 1 -- 172.21.15.97:0/3798363168 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4172647953 0 0) 0x7fa4d4004eb0 con 0x7fa4f41430c0 2024-03-25T18:25:24.663 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.661+0000 7fa4e2ffd640 1 -- 172.21.15.97:0/3798363168 --> v1:172.21.15.97:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa4f41d7f80 con 0x7fa4f41430c0 2024-03-25T18:25:24.664 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.661+0000 7fa4e2ffd640 1 -- 172.21.15.97:0/3798363168 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7fa4ec001f40 con 0x7fa4f41430c0 2024-03-25T18:25:24.664 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.661+0000 7fa4e2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:24.664 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.661+0000 7fa4e2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:25:24.664 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.661+0000 7fa4e2ffd640 10 monclient(hunting): dump: 2024-03-25T18:25:24.664 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:25:24.664 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:25:24.664 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:25:24.664 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:25:24.664 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:24.664 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:25:24.665 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:25:24.665 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:25:24.665 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:25:24.665 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:25:24.665 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.661+0000 7fa4e2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:25:24.665 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.662+0000 7fa4e2ffd640 1 -- 172.21.15.97:0/3798363168 <== mon.2 v1:172.21.15.97:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2833265393 0 0) 0x7fa4f41d7df0 con 0x7fa4f413f710 2024-03-25T18:25:24.665 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.662+0000 7fa4e2ffd640 1 -- 172.21.15.97:0/3798363168 --> v1:172.21.15.97:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa4d4004eb0 con 0x7fa4f413f710 2024-03-25T18:25:24.666 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.662+0000 7fa4e2ffd640 1 -- 172.21.15.97:0/3798363168 <== mon.2 v1:172.21.15.97:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7fa4e8001930 con 0x7fa4f413f710 2024-03-25T18:25:24.666 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.662+0000 7fa4e2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:24.666 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.662+0000 7fa4e2ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:25:24.666 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.662+0000 7fa4e2ffd640 10 monclient(hunting): dump: 2024-03-25T18:25:24.666 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:25:24.666 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:25:24.666 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:25:24.666 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:25:24.666 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:24.666 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:25:24.666 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:25:24.666 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:25:24.666 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:25:24.666 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:25:24.668 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.666+0000 7fa4e2ffd640 1 -- 172.21.15.97:0/3798363168 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2313823148 0 0) 0x7fa4f413b5c0 con 0x7fa4f40a98f0 2024-03-25T18:25:24.668 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.666+0000 7fa4e2ffd640 1 -- 172.21.15.97:0/3798363168 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa4f41d7df0 con 0x7fa4f40a98f0 2024-03-25T18:25:24.668 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.666+0000 7fa4e2ffd640 1 -- 172.21.15.97:0/3798363168 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7fa4e4001860 con 0x7fa4f40a98f0 2024-03-25T18:25:24.668 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.666+0000 7fa4e2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:24.668 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.666+0000 7fa4e2ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:25:24.668 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.666+0000 7fa4e2ffd640 10 monclient(hunting): dump: 2024-03-25T18:25:24.668 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:25:24.669 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:25:24.669 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:25:24.669 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:25:24.669 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:24.669 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:25:24.669 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:25:24.669 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:25:24.669 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:25:24.669 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:25:24.680 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.677+0000 7fa4e2ffd640 1 -- 172.21.15.97:0/3798363168 <== mon.0 v1:172.21.15.97:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1354454110 0 0) 0x7fa4f41d7f80 con 0x7fa4f41430c0 2024-03-25T18:25:24.680 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.677+0000 7fa4e2ffd640 1 -- 172.21.15.97:0/3798363168 >> v1:172.21.15.97:6790/0 conn(0x7fa4f413f710 legacy=0x7fa4f4133000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:24.680 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.677+0000 7fa4e2ffd640 1 -- 172.21.15.97:0/3798363168 >> v1:172.21.15.98:6789/0 conn(0x7fa4f40a98f0 legacy=0x7fa4f41d76e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:24.680 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.677+0000 7fa4e2ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:25:24.680 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.677+0000 7fa4e2ffd640 1 monclient: found mon.a 2024-03-25T18:25:24.680 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.677+0000 7fa4e2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:24.680 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.677+0000 7fa4e2ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:25:24.680 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.677+0000 7fa4e2ffd640 1 -- 172.21.15.97:0/3798363168 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4f413b750 con 0x7fa4f41430c0 2024-03-25T18:25:24.680 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.677+0000 7fa4e2ffd640 10 monclient: _finish_auth 0 2024-03-25T18:25:24.680 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.677+0000 7fa4e2ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:25:24.680 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.677+0000 7fa4e2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:24.681 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.677+0000 7fa4f8eb5640 5 monclient: authenticate success, global_id 4635 2024-03-25T18:25:24.681 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.677+0000 7fa4f8eb5640 10 monclient: _renew_subs 2024-03-25T18:25:24.681 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.677+0000 7fa4f8eb5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:25:24.681 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.677+0000 7fa4f8eb5640 1 -- 172.21.15.97:0/3798363168 --> v1:172.21.15.97:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4f4133710 con 0x7fa4f41430c0 2024-03-25T18:25:24.681 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.678+0000 7fa4f8eb5640 10 client.4635.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:25:24.681 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.678+0000 7fa4f8eb5640 10 monclient: _renew_subs 2024-03-25T18:25:24.681 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.678+0000 7fa4f8eb5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:25:24.681 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.678+0000 7fa4f8eb5640 1 -- 172.21.15.97:0/3798363168 --> v1:172.21.15.97:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4f41357a0 con 0x7fa4f41430c0 2024-03-25T18:25:24.681 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.678+0000 7fa4f8eb5640 1 librados: init done 2024-03-25T18:25:24.682 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.678+0000 7fa4f8eb5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:24.682 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.678+0000 7fa4f8eb5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:24.682 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.678+0000 7fa4f8eb5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:25:24.682 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.678+0000 7fa4f8eb5640 1 -- 172.21.15.97:0/3798363168 --> v1:172.21.15.97:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4f4138fd0 con 0x7fa4f41430c0 2024-03-25T18:25:24.995 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.993+0000 7fa4e2ffd640 1 -- 172.21.15.97:0/3798363168 <== mon.0 v1:172.21.15.97:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa4ec004e10 con 0x7fa4f41430c0 2024-03-25T18:25:24.995 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.993+0000 7fa4e2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:24.995 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.993+0000 7fa4e2ffd640 1 -- 172.21.15.97:0/3798363168 <== mon.0 v1:172.21.15.97:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7fa4ec004f80 con 0x7fa4f41430c0 2024-03-25T18:25:24.995 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.993+0000 7fa4e2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:24.995 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.993+0000 7fa4e2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:25:24.996 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.993+0000 7fa4e2ffd640 10 monclient: dump: 2024-03-25T18:25:24.996 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:25:24.996 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:25:24.996 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:25:24.996 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:25:24.996 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:24.996 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:25:24.996 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:25:24.996 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:25:24.996 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:25:24.996 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:25:24.996 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.993+0000 7fa4f0d99640 4 set_mon_vals no callback set 2024-03-25T18:25:24.997 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.994+0000 7fa4e2ffd640 1 -- 172.21.15.97:0/3798363168 <== mon.0 v1:172.21.15.97:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2377622137 0 0) 0x7fa4ec01af50 con 0x7fa4f41430c0 2024-03-25T18:25:24.997 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.995+0000 7fa4f259c640 10 monclient: get_auth_request con 0x7fa4d406e0f0 auth_method 0 2024-03-25T18:25:24.997 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:24.995+0000 7fa4e2ffd640 10 client.4635.objecter ms_handle_connect 0x7fa4d406e0f0 2024-03-25T18:25:25.002 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:25.000+0000 7fa4e2ffd640 1 -- 172.21.15.97:0/3798363168 <== mon.0 v1:172.21.15.97:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 430711918 0 0) 0x7fa4ec054a20 con 0x7fa4f41430c0 2024-03-25T18:25:25.002 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:25.000+0000 7fa4e2ffd640 10 client.4635.objecter ms_dispatch 0x7fa4f4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:25:25.002 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:25.000+0000 7fa4e2ffd640 3 client.4635.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:25:25.002 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:25.000+0000 7fa4e2ffd640 3 client.4635.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:25:25.002 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:25.000+0000 7fa4e2ffd640 20 client.4635.objecter dump_active .. 0 homeless 2024-03-25T18:25:25.002 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:25.000+0000 7fa4e2ffd640 1 -- 172.21.15.97:0/3798363168 <== mon.0 v1:172.21.15.97:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa4ec050530 con 0x7fa4f41430c0 2024-03-25T18:25:25.002 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:25.000+0000 7fa4e2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:25.003 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:25.000+0000 7fa4e2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:25:25.238 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:25.236+0000 7fa4f8eb5640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:25.238 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:25.236+0000 7fa4f8eb5640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:25.239 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:25.236+0000 7fa4f8eb5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:25:25.239 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:25.236+0000 7fa4f8eb5640 1 -- 172.21.15.97:0/3798363168 --> v1:172.21.15.97:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa4f4074cc0 con 0x7fa4f41430c0 2024-03-25T18:25:25.268 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:25.266+0000 7fa4e2ffd640 1 -- 172.21.15.97:0/3798363168 <== mon.0 v1:172.21.15.97:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2169427412) 0x7fa4ec052460 con 0x7fa4f41430c0 2024-03-25T18:25:25.268 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:25.266+0000 7fa4e2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:25.268 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:25.266+0000 7fa4e2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:25:25.268 INFO:teuthology.misc.health.smithi097.stdout: 2024-03-25T18:25:25.268 INFO:teuthology.misc.health.smithi097.stdout:{"epoch":10,"fsid":"4c3d472c-79b8-486b-9e3d-2738d4306754","created":"2024-03-25T18:16:14.737309+0000","modified":"2024-03-25T18:16:20.357492+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:20.357492+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cb5152b9-262f-45c3-94b8-18d0e81df558","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6daf8793-0838-473d-a7d6-9a0d2894c75d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"136f1493-3b9c-4f43-a2e7-641482d69dae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"397d538c-4f64-4daf-9e8b-662d09f91327","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4f1df08-663f-4bda-93f8-3878ab7c24b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c4b18d70-a3ce-4b15-9b70-eae5525bd761","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8dd4033a-de0e-4f8a-96dc-59f3c1f825b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0077722e-ffa6-4a3d-9891-929d9869b286","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:25:25.270 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:25.268+0000 7fa4f8eb5640 10 librados: watch_flush enter 2024-03-25T18:25:25.270 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:25.268+0000 7fa4f8eb5640 10 librados: watch_flush exit 2024-03-25T18:25:25.270 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:25.268+0000 7fa4f8eb5640 20 client.4635.objecter shutdown clearing up homeless session... 2024-03-25T18:25:25.271 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:25.268+0000 7fa4f8eb5640 10 client.4635.objecter successfully canceled tick 2024-03-25T18:25:25.271 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:25.268+0000 7fa4f8eb5640 1 -- 172.21.15.97:0/3798363168 >> v1:172.21.15.98:6800/800387573 conn(0x7fa4d406e0f0 legacy=0x7fa4d40705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:25.271 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:25.268+0000 7fa4f8eb5640 10 monclient: shutdown 2024-03-25T18:25:25.271 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:25.268+0000 7fa4f8eb5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:25.271 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:25.268+0000 7fa4f8eb5640 1 -- 172.21.15.97:0/3798363168 >> v1:172.21.15.97:6789/0 conn(0x7fa4f41430c0 legacy=0x7fa4f4135090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:25.271 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:25.268+0000 7fa4f8eb5640 1 -- 172.21.15.97:0/3798363168 shutdown_connections 2024-03-25T18:25:25.271 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:25.268+0000 7fa4f8eb5640 1 -- 172.21.15.97:0/3798363168 >> 172.21.15.97:0/3798363168 conn(0x7fa4f40b7ea0 msgr2=0x7fa4f4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:25.271 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:25.268+0000 7fa4f8eb5640 1 -- 172.21.15.97:0/3798363168 shutdown_connections 2024-03-25T18:25:25.271 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:25.268+0000 7fa4f8eb5640 1 -- 172.21.15.97:0/3798363168 wait complete. 2024-03-25T18:25:25.271 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:25.268+0000 7fa4f8eb5640 1 librados: shutdown 2024-03-25T18:25:25.283 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:25:32.090 DEBUG:teuthology.orchestra.run.smithi097:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:25:32.204 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.203+0000 7f999555d640 10 monclient: get_monmap_and_config 2024-03-25T18:25:32.205 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.203+0000 7f999555d640 10 monclient: build_initial_monmap 2024-03-25T18:25:32.205 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.203+0000 7f999555d640 10 monclient: monmap: 2024-03-25T18:25:32.205 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:25:32.205 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:25:32.205 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:25:32.204479+0000 2024-03-25T18:25:32.205 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:25:32.204479+0000 2024-03-25T18:25:32.205 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:32.205 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:25:32.205 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:25:32.205 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T18:25:32.205 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-b 2024-03-25T18:25:32.205 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:25:32.206 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.204+0000 7f999555d640 1 Processor -- start 2024-03-25T18:25:32.206 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.204+0000 7f999555d640 1 -- start start 2024-03-25T18:25:32.206 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.204+0000 7f999555d640 10 monclient: init 2024-03-25T18:25:32.207 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.205+0000 7f999555d640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:32.207 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.205+0000 7f999555d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:25:32.207 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.205+0000 7f999555d640 10 monclient(hunting): picked mon.noname-b con 0x7f99900a98f0 addr v1:172.21.15.97:6790/0 2024-03-25T18:25:32.207 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.205+0000 7f999555d640 10 monclient(hunting): picked mon.noname-a con 0x7f999013f710 addr v1:172.21.15.97:6789/0 2024-03-25T18:25:32.207 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.205+0000 7f999555d640 10 monclient(hunting): picked mon.noname-c con 0x7f99901430c0 addr v1:172.21.15.98:6789/0 2024-03-25T18:25:32.207 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.205+0000 7f999555d640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9990142520 con 0x7f999013f710 2024-03-25T18:25:32.207 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.205+0000 7f999555d640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99900b28a0 con 0x7f99901430c0 2024-03-25T18:25:32.207 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.205+0000 7f999555d640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99900b2a30 con 0x7f99900a98f0 2024-03-25T18:25:32.207 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.205+0000 7f999555d640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:32.207 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.205+0000 7f999555d640 10 monclient(hunting): authenticate will time out at 8144.481215s 2024-03-25T18:25:32.210 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.208+0000 7f998e7fc640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f999013f710 0x7f99901429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:51288/0 (socket says 172.21.15.97:51288) 2024-03-25T18:25:32.210 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.208+0000 7f998e7fc640 1 -- 172.21.15.97:0/1409327051 learned_addr learned my addr 172.21.15.97:0/1409327051 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:25:32.227 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.226+0000 7f998dffb640 1 -- 172.21.15.97:0/1409327051 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 803546766 0 0) 0x7f99900b2a30 con 0x7f99900a98f0 2024-03-25T18:25:32.227 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.226+0000 7f998dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:25:32.228 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.226+0000 7f998dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:32.228 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.226+0000 7f998dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:32.228 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.226+0000 7f998dffb640 10 monclient(hunting): my global_id is 4625 2024-03-25T18:25:32.228 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.226+0000 7f998dffb640 1 -- 172.21.15.97:0/1409327051 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9978003520 con 0x7f99900a98f0 2024-03-25T18:25:32.233 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.231+0000 7f998dffb640 1 -- 172.21.15.97:0/1409327051 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1906540602 0 0) 0x7f99900b28a0 con 0x7f99901430c0 2024-03-25T18:25:32.233 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.231+0000 7f998dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:25:32.233 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.231+0000 7f998dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:32.233 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.231+0000 7f998dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:32.233 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.231+0000 7f998dffb640 10 monclient(hunting): my global_id is 4642 2024-03-25T18:25:32.233 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.231+0000 7f998dffb640 1 -- 172.21.15.97:0/1409327051 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99900b2a30 con 0x7f99901430c0 2024-03-25T18:25:32.237 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.235+0000 7f998dffb640 1 -- 172.21.15.97:0/1409327051 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3343111035 0 0) 0x7f9990142520 con 0x7f999013f710 2024-03-25T18:25:32.237 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.235+0000 7f998dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:25:32.237 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.235+0000 7f998dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:32.237 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.235+0000 7f998dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:32.237 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.235+0000 7f998dffb640 10 monclient(hunting): my global_id is 4638 2024-03-25T18:25:32.237 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.235+0000 7f998dffb640 1 -- 172.21.15.97:0/1409327051 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99900b28a0 con 0x7f999013f710 2024-03-25T18:25:32.240 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.238+0000 7f998dffb640 1 -- 172.21.15.97:0/1409327051 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3383218582 0 0) 0x7f99900b2a30 con 0x7f99901430c0 2024-03-25T18:25:32.240 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.238+0000 7f998dffb640 1 -- 172.21.15.97:0/1409327051 >> v1:172.21.15.97:6790/0 conn(0x7f99900a98f0 legacy=0x7f99900a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:32.240 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.238+0000 7f998dffb640 1 -- 172.21.15.97:0/1409327051 >> v1:172.21.15.97:6789/0 conn(0x7f999013f710 legacy=0x7f99901429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:32.240 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.238+0000 7f998dffb640 10 monclient: _finish_hunting 0 2024-03-25T18:25:32.240 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.238+0000 7f998dffb640 1 monclient: found mon.noname-c 2024-03-25T18:25:32.241 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.238+0000 7f998dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:32.241 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.238+0000 7f998dffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.98:6789/0 2024-03-25T18:25:32.241 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.238+0000 7f998dffb640 1 -- 172.21.15.97:0/1409327051 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99900b44c0 con 0x7f99901430c0 2024-03-25T18:25:32.241 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.238+0000 7f998dffb640 10 monclient: _finish_auth 0 2024-03-25T18:25:32.241 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.238+0000 7f998dffb640 10 monclient: _check_auth_tickets 2024-03-25T18:25:32.241 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.238+0000 7f998dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:32.241 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.238+0000 7f999555d640 5 monclient: authenticate success, global_id 4642 2024-03-25T18:25:32.241 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.238+0000 7f999555d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:25:32.241 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.238+0000 7f998dffb640 1 -- 172.21.15.97:0/1409327051 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f99880019f0 con 0x7f99901430c0 2024-03-25T18:25:32.241 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.238+0000 7f998dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:32.241 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.238+0000 7f998dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:25:32.241 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.238+0000 7f998dffb640 10 monclient: dump: 2024-03-25T18:25:32.241 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:25:32.241 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:25:32.241 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:25:32.241 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:25:32.242 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:32.242 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:25:32.242 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:25:32.242 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:25:32.242 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:25:32.242 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:25:32.242 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.238+0000 7f999555d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:25:32.248 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.247+0000 7f998dffb640 1 -- 172.21.15.97:0/1409327051 <== mon.1 v1:172.21.15.98:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9988001b50 con 0x7f99901430c0 2024-03-25T18:25:32.248 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.247+0000 7f998dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:32.249 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.247+0000 7f998dffb640 1 -- 172.21.15.97:0/1409327051 <== mon.1 v1:172.21.15.98:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f9988001f60 con 0x7f99901430c0 2024-03-25T18:25:32.249 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.247+0000 7f998dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:32.249 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.247+0000 7f998dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:25:32.249 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.247+0000 7f998dffb640 10 monclient: dump: 2024-03-25T18:25:32.249 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:25:32.249 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:25:32.249 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:25:32.249 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:25:32.249 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:32.249 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:25:32.249 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:25:32.249 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:25:32.249 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:25:32.249 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:25:32.250 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.247+0000 7f999555d640 10 monclient: get_monmap_and_config success 2024-03-25T18:25:32.250 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.247+0000 7f999555d640 4 set_mon_vals no callback set 2024-03-25T18:25:32.251 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.248+0000 7f999555d640 10 monclient: shutdown 2024-03-25T18:25:32.251 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.248+0000 7f999555d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:32.251 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.248+0000 7f999555d640 1 -- 172.21.15.97:0/1409327051 >> v1:172.21.15.98:6789/0 conn(0x7f99901430c0 legacy=0x7f999014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:32.251 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.248+0000 7f999555d640 1 -- 172.21.15.97:0/1409327051 shutdown_connections 2024-03-25T18:25:32.251 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.248+0000 7f999555d640 1 -- 172.21.15.97:0/1409327051 >> 172.21.15.97:0/1409327051 conn(0x7f99900b7ea0 msgr2=0x7f99900b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:32.251 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.248+0000 7f999555d640 1 -- 172.21.15.97:0/1409327051 shutdown_connections 2024-03-25T18:25:32.251 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.248+0000 7f999555d640 1 -- 172.21.15.97:0/1409327051 wait complete. 2024-03-25T18:25:32.251 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.248+0000 7f999555d640 10 monclient: build_initial_monmap 2024-03-25T18:25:32.251 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.248+0000 7f999555d640 10 monclient: monmap: 2024-03-25T18:25:32.252 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:25:32.252 INFO:teuthology.misc.health.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:25:32.252 INFO:teuthology.misc.health.smithi097.stderr:last_changed 0.000000 2024-03-25T18:25:32.252 INFO:teuthology.misc.health.smithi097.stderr:created 0.000000 2024-03-25T18:25:32.252 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:32.252 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 1 2024-03-25T18:25:32.252 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:25:32.252 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T18:25:32.252 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-c 2024-03-25T18:25:32.252 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:25:32.252 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.249+0000 7f999555d640 1 librados: starting msgr at 2024-03-25T18:25:32.252 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.249+0000 7f999555d640 1 librados: starting objecter 2024-03-25T18:25:32.252 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.249+0000 7f999555d640 1 Processor -- start 2024-03-25T18:25:32.252 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.249+0000 7f999555d640 1 -- start start 2024-03-25T18:25:32.252 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.249+0000 7f999555d640 1 librados: setting wanted keys 2024-03-25T18:25:32.253 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.249+0000 7f999555d640 1 librados: calling monclient init 2024-03-25T18:25:32.253 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.249+0000 7f999555d640 10 monclient: init 2024-03-25T18:25:32.253 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.250+0000 7f999555d640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:32.253 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.250+0000 7f999555d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:25:32.254 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.250+0000 7f999555d640 10 monclient(hunting): picked mon.noname-b con 0x7f99900a98f0 addr v1:172.21.15.98:6789/0 2024-03-25T18:25:32.254 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.250+0000 7f999555d640 10 monclient(hunting): picked mon.noname-c con 0x7f999013f710 addr v1:172.21.15.97:6790/0 2024-03-25T18:25:32.254 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.250+0000 7f999555d640 10 monclient(hunting): picked mon.noname-a con 0x7f99901430c0 addr v1:172.21.15.97:6789/0 2024-03-25T18:25:32.254 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.250+0000 7f999555d640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99901388a0 con 0x7f99901430c0 2024-03-25T18:25:32.254 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.250+0000 7f999555d640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99901349e0 con 0x7f99900a98f0 2024-03-25T18:25:32.254 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.250+0000 7f999555d640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9990138d30 con 0x7f999013f710 2024-03-25T18:25:32.254 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.250+0000 7f999555d640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:32.254 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.250+0000 7f999555d640 10 monclient(hunting): authenticate will time out at 8144.526147s 2024-03-25T18:25:32.256 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.254+0000 7f998f7fe640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f99901430c0 0x7f99901342d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:51342/0 (socket says 172.21.15.97:51342) 2024-03-25T18:25:32.256 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.254+0000 7f998f7fe640 1 -- 172.21.15.97:0/1917655240 learned_addr learned my addr 172.21.15.97:0/1917655240 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:25:32.257 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.256+0000 7f99777fe640 10 client.?.objecter ms_handle_connect 0x7f99901430c0 2024-03-25T18:25:32.257 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.256+0000 7f99777fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:32.266 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.265+0000 7f99777fe640 10 client.?.objecter ms_handle_connect 0x7f999013f710 2024-03-25T18:25:32.266 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.265+0000 7f99777fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:32.267 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.265+0000 7f99777fe640 1 -- 172.21.15.97:0/1917655240 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2122647289 0 0) 0x7f99901388a0 con 0x7f99901430c0 2024-03-25T18:25:32.267 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.265+0000 7f99777fe640 10 monclient(hunting): _negotiate 2024-03-25T18:25:32.267 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.265+0000 7f99777fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:32.267 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.265+0000 7f99777fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:32.267 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.265+0000 7f99777fe640 10 monclient(hunting): my global_id is 4641 2024-03-25T18:25:32.267 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.265+0000 7f99777fe640 1 -- 172.21.15.97:0/1917655240 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f997c004eb0 con 0x7f99901430c0 2024-03-25T18:25:32.276 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.275+0000 7f99777fe640 10 client.?.objecter ms_handle_connect 0x7f99900a98f0 2024-03-25T18:25:32.276 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.275+0000 7f99777fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:32.278 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.276+0000 7f99777fe640 1 -- 172.21.15.97:0/1917655240 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 729030490 0 0) 0x7f9990138d30 con 0x7f999013f710 2024-03-25T18:25:32.278 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.276+0000 7f99777fe640 10 monclient(hunting): _negotiate 2024-03-25T18:25:32.278 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.276+0000 7f99777fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:32.278 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.276+0000 7f99777fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:32.278 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.276+0000 7f99777fe640 10 monclient(hunting): my global_id is 4628 2024-03-25T18:25:32.278 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.276+0000 7f99777fe640 1 -- 172.21.15.97:0/1917655240 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99901388a0 con 0x7f999013f710 2024-03-25T18:25:32.288 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.287+0000 7f99777fe640 1 -- 172.21.15.97:0/1917655240 <== mon.2 v1:172.21.15.97:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3512958706 0 0) 0x7f99901388a0 con 0x7f999013f710 2024-03-25T18:25:32.288 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.287+0000 7f99777fe640 1 -- 172.21.15.97:0/1917655240 --> v1:172.21.15.97:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9990138d30 con 0x7f999013f710 2024-03-25T18:25:32.288 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.287+0000 7f99777fe640 1 -- 172.21.15.97:0/1917655240 <== mon.2 v1:172.21.15.97:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f9980001870 con 0x7f999013f710 2024-03-25T18:25:32.288 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.287+0000 7f99777fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:32.289 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.287+0000 7f99777fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:25:32.289 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.287+0000 7f99777fe640 10 monclient(hunting): dump: 2024-03-25T18:25:32.289 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:25:32.289 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:25:32.289 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:25:32.289 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:25:32.289 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:32.289 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:25:32.289 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:25:32.289 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:25:32.290 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:25:32.290 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:25:32.290 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.287+0000 7f99777fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:25:32.290 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.288+0000 7f99777fe640 1 -- 172.21.15.97:0/1917655240 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1567733567 0 0) 0x7f997c004eb0 con 0x7f99901430c0 2024-03-25T18:25:32.290 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.288+0000 7f99777fe640 1 -- 172.21.15.97:0/1917655240 --> v1:172.21.15.97:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f99901388a0 con 0x7f99901430c0 2024-03-25T18:25:32.291 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.288+0000 7f99777fe640 1 -- 172.21.15.97:0/1917655240 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f9988001fa0 con 0x7f99901430c0 2024-03-25T18:25:32.291 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.288+0000 7f99777fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:32.291 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.288+0000 7f99777fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:25:32.291 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.288+0000 7f99777fe640 10 monclient(hunting): dump: 2024-03-25T18:25:32.291 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:25:32.291 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:25:32.291 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:25:32.291 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:25:32.291 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:32.291 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:25:32.291 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:25:32.291 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:25:32.291 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:25:32.292 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:25:32.297 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.295+0000 7f99777fe640 1 -- 172.21.15.97:0/1917655240 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2605456022 0 0) 0x7f99901349e0 con 0x7f99900a98f0 2024-03-25T18:25:32.297 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.295+0000 7f99777fe640 10 monclient(hunting): _negotiate 2024-03-25T18:25:32.297 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.295+0000 7f99777fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:32.297 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.295+0000 7f99777fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:32.297 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.295+0000 7f99777fe640 10 monclient(hunting): my global_id is 4645 2024-03-25T18:25:32.297 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.295+0000 7f99777fe640 1 -- 172.21.15.97:0/1917655240 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f997c004eb0 con 0x7f99900a98f0 2024-03-25T18:25:32.299 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.297+0000 7f99777fe640 1 -- 172.21.15.97:0/1917655240 <== mon.2 v1:172.21.15.97:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2512127322 0 0) 0x7f9990138d30 con 0x7f999013f710 2024-03-25T18:25:32.299 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.297+0000 7f99777fe640 1 -- 172.21.15.97:0/1917655240 >> v1:172.21.15.98:6789/0 conn(0x7f99900a98f0 legacy=0x7f99901da880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:32.299 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.297+0000 7f99777fe640 1 -- 172.21.15.97:0/1917655240 >> v1:172.21.15.97:6789/0 conn(0x7f99901430c0 legacy=0x7f99901342d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:32.299 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.297+0000 7f99777fe640 10 monclient: _finish_hunting 0 2024-03-25T18:25:32.300 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.297+0000 7f99777fe640 1 monclient: found mon.c 2024-03-25T18:25:32.300 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.297+0000 7f99777fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:32.300 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.297+0000 7f99777fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.97:6790/0 2024-03-25T18:25:32.300 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.297+0000 7f99777fe640 1 -- 172.21.15.97:0/1917655240 --> v1:172.21.15.97:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99900aba50 con 0x7f999013f710 2024-03-25T18:25:32.300 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.297+0000 7f99777fe640 10 monclient: _finish_auth 0 2024-03-25T18:25:32.300 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.297+0000 7f99777fe640 10 monclient: _check_auth_tickets 2024-03-25T18:25:32.300 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.297+0000 7f99777fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:32.300 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.297+0000 7f999555d640 5 monclient: authenticate success, global_id 4628 2024-03-25T18:25:32.300 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.297+0000 7f999555d640 10 monclient: _renew_subs 2024-03-25T18:25:32.300 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.297+0000 7f999555d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.97:6790/0 2024-03-25T18:25:32.300 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.297+0000 7f999555d640 1 -- 172.21.15.97:0/1917655240 --> v1:172.21.15.97:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99900b1770 con 0x7f999013f710 2024-03-25T18:25:32.300 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.297+0000 7f999555d640 10 client.4628.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:25:32.300 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.297+0000 7f999555d640 10 monclient: _renew_subs 2024-03-25T18:25:32.300 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.297+0000 7f999555d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.97:6790/0 2024-03-25T18:25:32.300 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.297+0000 7f999555d640 1 -- 172.21.15.97:0/1917655240 --> v1:172.21.15.97:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9990137ba0 con 0x7f999013f710 2024-03-25T18:25:32.300 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.297+0000 7f999555d640 1 librados: init done 2024-03-25T18:25:32.301 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.298+0000 7f999555d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:32.301 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.298+0000 7f999555d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:32.301 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.298+0000 7f999555d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.97:6790/0 2024-03-25T18:25:32.301 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.298+0000 7f999555d640 1 -- 172.21.15.97:0/1917655240 --> v1:172.21.15.97:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99900627b0 con 0x7f999013f710 2024-03-25T18:25:32.606 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.604+0000 7f99777fe640 1 -- 172.21.15.97:0/1917655240 <== mon.2 v1:172.21.15.97:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9980001cc0 con 0x7f999013f710 2024-03-25T18:25:32.606 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.604+0000 7f99777fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:32.606 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.604+0000 7f99777fe640 1 -- 172.21.15.97:0/1917655240 <== mon.2 v1:172.21.15.97:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f9980004a40 con 0x7f999013f710 2024-03-25T18:25:32.607 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.604+0000 7f99777fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:32.607 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.604+0000 7f99777fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:25:32.607 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.604+0000 7f99777fe640 10 monclient: dump: 2024-03-25T18:25:32.607 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:25:32.607 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:25:32.607 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:25:32.607 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:25:32.607 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:32.607 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:25:32.607 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:25:32.607 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:25:32.607 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:25:32.607 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:25:32.607 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.604+0000 7f998d7fa640 4 set_mon_vals no callback set 2024-03-25T18:25:32.608 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.605+0000 7f99777fe640 1 -- 172.21.15.97:0/1917655240 <== mon.2 v1:172.21.15.97:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2377622137 0 0) 0x7f9980004c60 con 0x7f999013f710 2024-03-25T18:25:32.608 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.606+0000 7f998effd640 10 monclient: get_auth_request con 0x7f997c06df50 auth_method 0 2024-03-25T18:25:32.608 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.606+0000 7f99777fe640 10 client.4628.objecter ms_handle_connect 0x7f997c06df50 2024-03-25T18:25:32.614 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.612+0000 7f99777fe640 1 -- 172.21.15.97:0/1917655240 <== mon.2 v1:172.21.15.97:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 430711918 0 0) 0x7f9980054750 con 0x7f999013f710 2024-03-25T18:25:32.614 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.612+0000 7f99777fe640 10 client.4628.objecter ms_dispatch 0x7f9990000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:25:32.614 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.612+0000 7f99777fe640 3 client.4628.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:25:32.614 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.612+0000 7f99777fe640 3 client.4628.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:25:32.614 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.612+0000 7f99777fe640 20 client.4628.objecter dump_active .. 0 homeless 2024-03-25T18:25:32.614 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.612+0000 7f99777fe640 1 -- 172.21.15.97:0/1917655240 <== mon.2 v1:172.21.15.97:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9980050260 con 0x7f999013f710 2024-03-25T18:25:32.615 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.612+0000 7f99777fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:32.615 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.612+0000 7f99777fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:25:32.854 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.852+0000 7f999555d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:32.854 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.852+0000 7f999555d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:32.854 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.852+0000 7f999555d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.97:6790/0 2024-03-25T18:25:32.854 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.852+0000 7f999555d640 1 -- 172.21.15.97:0/1917655240 --> v1:172.21.15.97:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9990141a00 con 0x7f999013f710 2024-03-25T18:25:32.897 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.895+0000 7f99777fe640 1 -- 172.21.15.97:0/1917655240 <== mon.2 v1:172.21.15.97:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2169427412) 0x7f9980052190 con 0x7f999013f710 2024-03-25T18:25:32.897 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.895+0000 7f99777fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:32.897 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.895+0000 7f99777fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:25:32.897 INFO:teuthology.misc.health.smithi097.stdout: 2024-03-25T18:25:32.897 INFO:teuthology.misc.health.smithi097.stdout:{"epoch":10,"fsid":"4c3d472c-79b8-486b-9e3d-2738d4306754","created":"2024-03-25T18:16:14.737309+0000","modified":"2024-03-25T18:16:20.357492+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:20.357492+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cb5152b9-262f-45c3-94b8-18d0e81df558","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6daf8793-0838-473d-a7d6-9a0d2894c75d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"136f1493-3b9c-4f43-a2e7-641482d69dae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"397d538c-4f64-4daf-9e8b-662d09f91327","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4f1df08-663f-4bda-93f8-3878ab7c24b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c4b18d70-a3ce-4b15-9b70-eae5525bd761","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8dd4033a-de0e-4f8a-96dc-59f3c1f825b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0077722e-ffa6-4a3d-9891-929d9869b286","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:25:32.900 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.897+0000 7f999555d640 10 librados: watch_flush enter 2024-03-25T18:25:32.900 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.897+0000 7f999555d640 10 librados: watch_flush exit 2024-03-25T18:25:32.900 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.897+0000 7f999555d640 20 client.4628.objecter shutdown clearing up homeless session... 2024-03-25T18:25:32.900 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.897+0000 7f999555d640 10 client.4628.objecter successfully canceled tick 2024-03-25T18:25:32.900 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.898+0000 7f999555d640 1 -- 172.21.15.97:0/1917655240 >> v1:172.21.15.98:6800/800387573 conn(0x7f997c06df50 legacy=0x7f997c070410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:32.900 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.898+0000 7f999555d640 10 monclient: shutdown 2024-03-25T18:25:32.900 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.898+0000 7f999555d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:32.900 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.898+0000 7f999555d640 1 -- 172.21.15.97:0/1917655240 >> v1:172.21.15.97:6790/0 conn(0x7f999013f710 legacy=0x7f9990133bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:32.900 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.898+0000 7f999555d640 1 -- 172.21.15.97:0/1917655240 shutdown_connections 2024-03-25T18:25:32.900 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.898+0000 7f999555d640 1 -- 172.21.15.97:0/1917655240 >> 172.21.15.97:0/1917655240 conn(0x7f99900b7ea0 msgr2=0x7f9990074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:32.900 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.898+0000 7f999555d640 1 -- 172.21.15.97:0/1917655240 shutdown_connections 2024-03-25T18:25:32.901 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.898+0000 7f999555d640 1 -- 172.21.15.97:0/1917655240 wait complete. 2024-03-25T18:25:32.901 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:32.898+0000 7f999555d640 1 librados: shutdown 2024-03-25T18:25:32.911 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:25:38.615 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:09:40.653 34931== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:38.678 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:09:40.717 34931== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:38.710 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:09:40.748 34931== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:38.715 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:09:40.758 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:38.721 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:09:40.760 34935== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:38.723 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:09:40.766 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:38.727 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:09:40.765 34935== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:38.739 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:09:40.783 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:38.752 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:09:40.791 34935== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:39.718 DEBUG:teuthology.orchestra.run.smithi097:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:25:39.830 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:39.828+0000 7eff63fd0640 10 monclient: get_monmap_and_config 2024-03-25T18:25:39.830 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:39.828+0000 7eff63fd0640 10 monclient: build_initial_monmap 2024-03-25T18:25:39.830 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:39.828+0000 7eff63fd0640 10 monclient: monmap: 2024-03-25T18:25:39.831 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:25:39.831 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:25:39.831 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:25:39.830137+0000 2024-03-25T18:25:39.831 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:25:39.830137+0000 2024-03-25T18:25:39.831 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:39.831 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:25:39.831 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:25:39.831 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T18:25:39.831 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-b 2024-03-25T18:25:39.831 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:25:39.832 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:39.829+0000 7eff63fd0640 1 Processor -- start 2024-03-25T18:25:39.832 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:39.829+0000 7eff63fd0640 1 -- start start 2024-03-25T18:25:39.833 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:39.829+0000 7eff63fd0640 10 monclient: init 2024-03-25T18:25:39.833 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:39.830+0000 7eff63fd0640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:39.833 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:39.830+0000 7eff63fd0640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:25:39.833 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:39.830+0000 7eff63fd0640 10 monclient(hunting): picked mon.noname-b con 0x7eff5c0b4d60 addr v1:172.21.15.97:6790/0 2024-03-25T18:25:39.833 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:39.830+0000 7eff63fd0640 10 monclient(hunting): picked mon.noname-c con 0x7eff5c0b1730 addr v1:172.21.15.98:6789/0 2024-03-25T18:25:39.833 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:39.830+0000 7eff63fd0640 10 monclient(hunting): picked mon.noname-a con 0x7eff5c14be90 addr v1:172.21.15.97:6789/0 2024-03-25T18:25:39.833 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:39.830+0000 7eff63fd0640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff5c0af940 con 0x7eff5c14be90 2024-03-25T18:25:39.833 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:39.830+0000 7eff63fd0640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff5c0b0a70 con 0x7eff5c0b1730 2024-03-25T18:25:39.833 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:39.830+0000 7eff63fd0640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff5c0b0d50 con 0x7eff5c0b4d60 2024-03-25T18:25:39.833 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:39.830+0000 7eff63fd0640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:39.833 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:39.830+0000 7eff63fd0640 10 monclient(hunting): authenticate will time out at 8152.106395s 2024-03-25T18:25:39.844 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:39.842+0000 7eff6256d640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7eff5c14be90 0x7eff5c14e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:55880/0 (socket says 172.21.15.97:55880) 2024-03-25T18:25:39.844 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:39.842+0000 7eff6256d640 1 -- 172.21.15.97:0/1956259004 learned_addr learned my addr 172.21.15.97:0/1956259004 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:25:39.852 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:39.850+0000 7eff60d6a640 1 -- 172.21.15.97:0/1956259004 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1241340660 0 0) 0x7eff5c0af940 con 0x7eff5c14be90 2024-03-25T18:25:39.852 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:39.850+0000 7eff60d6a640 10 monclient(hunting): _negotiate 2024-03-25T18:25:39.852 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:39.850+0000 7eff60d6a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:39.852 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:39.850+0000 7eff60d6a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:39.852 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:39.850+0000 7eff60d6a640 10 monclient(hunting): my global_id is 4644 2024-03-25T18:25:39.852 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:39.850+0000 7eff60d6a640 1 -- 172.21.15.97:0/1956259004 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff48003520 con 0x7eff5c14be90 2024-03-25T18:25:39.860 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:39.859+0000 7eff60d6a640 1 -- 172.21.15.97:0/1956259004 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 250331923 0 0) 0x7eff5c0b0a70 con 0x7eff5c0b1730 2024-03-25T18:25:39.861 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:39.859+0000 7eff60d6a640 10 monclient(hunting): _negotiate 2024-03-25T18:25:39.861 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:39.859+0000 7eff60d6a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:39.861 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:39.859+0000 7eff60d6a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:39.861 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:39.859+0000 7eff60d6a640 10 monclient(hunting): my global_id is 4648 2024-03-25T18:25:39.861 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:39.859+0000 7eff60d6a640 1 -- 172.21.15.97:0/1956259004 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff5c0af940 con 0x7eff5c0b1730 2024-03-25T18:25:39.862 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:39.860+0000 7eff60d6a640 1 -- 172.21.15.97:0/1956259004 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2497587260 0 0) 0x7eff5c0b0d50 con 0x7eff5c0b4d60 2024-03-25T18:25:39.862 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:39.860+0000 7eff60d6a640 10 monclient(hunting): _negotiate 2024-03-25T18:25:39.862 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:39.860+0000 7eff60d6a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:39.862 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:39.860+0000 7eff60d6a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:39.862 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:39.860+0000 7eff60d6a640 10 monclient(hunting): my global_id is 4631 2024-03-25T18:25:39.862 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:39.860+0000 7eff60d6a640 1 -- 172.21.15.97:0/1956259004 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff5c0b0a70 con 0x7eff5c0b4d60 2024-03-25T18:25:39.863 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:39.862+0000 7eff60d6a640 1 -- 172.21.15.97:0/1956259004 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1252149653 0 0) 0x7eff48003520 con 0x7eff5c14be90 2024-03-25T18:25:39.864 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:39.862+0000 7eff60d6a640 1 -- 172.21.15.97:0/1956259004 >> v1:172.21.15.97:6790/0 conn(0x7eff5c0b4d60 legacy=0x7eff5c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:39.864 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:39.862+0000 7eff60d6a640 1 -- 172.21.15.97:0/1956259004 >> v1:172.21.15.98:6789/0 conn(0x7eff5c0b1730 legacy=0x7eff5c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:39.864 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:39.862+0000 7eff60d6a640 10 monclient: _finish_hunting 0 2024-03-25T18:25:39.864 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:39.862+0000 7eff60d6a640 1 monclient: found mon.noname-a 2024-03-25T18:25:39.864 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:39.862+0000 7eff60d6a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:39.864 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:39.862+0000 7eff60d6a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.97:6789/0 2024-03-25T18:25:39.864 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:39.862+0000 7eff60d6a640 1 -- 172.21.15.97:0/1956259004 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff5c0b1030 con 0x7eff5c14be90 2024-03-25T18:25:39.864 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:39.862+0000 7eff60d6a640 10 monclient: _finish_auth 0 2024-03-25T18:25:39.864 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:39.862+0000 7eff60d6a640 10 monclient: _check_auth_tickets 2024-03-25T18:25:39.864 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:39.862+0000 7eff60d6a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:39.864 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:39.862+0000 7eff60d6a640 1 -- 172.21.15.97:0/1956259004 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7eff54001a10 con 0x7eff5c14be90 2024-03-25T18:25:39.864 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:39.862+0000 7eff60d6a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:39.864 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:39.862+0000 7eff60d6a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:25:39.865 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:39.862+0000 7eff60d6a640 10 monclient: dump: 2024-03-25T18:25:39.865 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:25:39.865 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:25:39.865 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:25:39.865 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:25:39.865 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:39.865 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:25:39.865 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:25:39.865 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:25:39.865 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:25:39.865 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:25:39.865 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:39.862+0000 7eff63fd0640 5 monclient: authenticate success, global_id 4644 2024-03-25T18:25:39.865 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:39.862+0000 7eff63fd0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:25:39.870 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:39.868+0000 7eff60d6a640 1 -- 172.21.15.97:0/1956259004 <== mon.0 v1:172.21.15.97:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7eff54001b70 con 0x7eff5c14be90 2024-03-25T18:25:39.870 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:39.868+0000 7eff60d6a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:39.870 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:39.868+0000 7eff63fd0640 10 monclient: get_monmap_and_config success 2024-03-25T18:25:39.870 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:39.868+0000 7eff63fd0640 4 set_mon_vals no callback set 2024-03-25T18:25:39.871 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:39.869+0000 7eff63fd0640 10 monclient: shutdown 2024-03-25T18:25:39.871 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:39.869+0000 7eff63fd0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:39.871 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:39.869+0000 7eff63fd0640 1 -- 172.21.15.97:0/1956259004 >> v1:172.21.15.97:6789/0 conn(0x7eff5c14be90 legacy=0x7eff5c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:39.871 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:39.869+0000 7eff63fd0640 1 -- 172.21.15.97:0/1956259004 shutdown_connections 2024-03-25T18:25:39.871 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:39.869+0000 7eff63fd0640 1 -- 172.21.15.97:0/1956259004 >> 172.21.15.97:0/1956259004 conn(0x7eff5c0617a0 msgr2=0x7eff5c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:39.871 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:39.869+0000 7eff63fd0640 1 -- 172.21.15.97:0/1956259004 shutdown_connections 2024-03-25T18:25:39.871 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:39.869+0000 7eff63fd0640 1 -- 172.21.15.97:0/1956259004 wait complete. 2024-03-25T18:25:39.872 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:39.870+0000 7eff63fd0640 10 monclient: build_initial_monmap 2024-03-25T18:25:39.872 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:39.870+0000 7eff63fd0640 10 monclient: monmap: 2024-03-25T18:25:39.872 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:25:39.872 INFO:teuthology.misc.health.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:25:39.872 INFO:teuthology.misc.health.smithi097.stderr:last_changed 0.000000 2024-03-25T18:25:39.872 INFO:teuthology.misc.health.smithi097.stderr:created 0.000000 2024-03-25T18:25:39.872 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:39.872 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 1 2024-03-25T18:25:39.873 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:25:39.873 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T18:25:39.873 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-c 2024-03-25T18:25:39.873 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:25:39.873 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:39.870+0000 7eff63fd0640 1 librados: starting msgr at 2024-03-25T18:25:39.873 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:39.870+0000 7eff63fd0640 1 librados: starting objecter 2024-03-25T18:25:39.873 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:39.870+0000 7eff63fd0640 1 Processor -- start 2024-03-25T18:25:39.873 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:39.870+0000 7eff63fd0640 1 -- start start 2024-03-25T18:25:39.873 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:39.870+0000 7eff63fd0640 1 librados: setting wanted keys 2024-03-25T18:25:39.873 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:39.870+0000 7eff63fd0640 1 librados: calling monclient init 2024-03-25T18:25:39.873 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:39.870+0000 7eff63fd0640 10 monclient: init 2024-03-25T18:25:39.874 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:39.871+0000 7eff63fd0640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:39.874 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:39.871+0000 7eff63fd0640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:25:39.874 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:39.871+0000 7eff63fd0640 10 monclient(hunting): picked mon.noname-a con 0x7eff5c0b1730 addr v1:172.21.15.97:6789/0 2024-03-25T18:25:39.874 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:39.871+0000 7eff63fd0640 10 monclient(hunting): picked mon.noname-c con 0x7eff5c0b4d60 addr v1:172.21.15.97:6790/0 2024-03-25T18:25:39.874 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:39.871+0000 7eff63fd0640 10 monclient(hunting): picked mon.noname-b con 0x7eff5c14be90 addr v1:172.21.15.98:6789/0 2024-03-25T18:25:39.875 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:39.871+0000 7eff63fd0640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff5c1d7960 con 0x7eff5c0b1730 2024-03-25T18:25:39.875 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:39.871+0000 7eff63fd0640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff5c1d8500 con 0x7eff5c14be90 2024-03-25T18:25:39.875 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:39.871+0000 7eff63fd0640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff5c138850 con 0x7eff5c0b4d60 2024-03-25T18:25:39.875 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:39.871+0000 7eff63fd0640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:39.875 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:39.871+0000 7eff63fd0640 10 monclient(hunting): authenticate will time out at 8152.147420s 2024-03-25T18:25:39.883 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:39.881+0000 7eff61d6c640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7eff5c0b1730 0x7eff5c1405f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:55928/0 (socket says 172.21.15.97:55928) 2024-03-25T18:25:39.883 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:39.882+0000 7eff61d6c640 1 -- 172.21.15.97:0/1032344487 learned_addr learned my addr 172.21.15.97:0/1032344487 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:25:39.885 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:39.884+0000 7eff527fc640 10 client.?.objecter ms_handle_connect 0x7eff5c0b1730 2024-03-25T18:25:39.885 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:39.884+0000 7eff527fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:39.894 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:39.892+0000 7eff527fc640 10 client.?.objecter ms_handle_connect 0x7eff5c0b4d60 2024-03-25T18:25:39.894 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:39.892+0000 7eff527fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:39.900 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:39.898+0000 7eff527fc640 1 -- 172.21.15.97:0/1032344487 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 409668884 0 0) 0x7eff5c1d7960 con 0x7eff5c0b1730 2024-03-25T18:25:39.900 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:39.898+0000 7eff527fc640 10 monclient(hunting): _negotiate 2024-03-25T18:25:39.900 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:39.898+0000 7eff527fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:39.900 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:39.898+0000 7eff527fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:39.900 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:39.898+0000 7eff527fc640 10 monclient(hunting): my global_id is 4647 2024-03-25T18:25:39.900 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:39.898+0000 7eff527fc640 1 -- 172.21.15.97:0/1032344487 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff40004eb0 con 0x7eff5c0b1730 2024-03-25T18:25:39.901 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:39.899+0000 7eff527fc640 1 -- 172.21.15.97:0/1032344487 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2713200126 0 0) 0x7eff5c138850 con 0x7eff5c0b4d60 2024-03-25T18:25:39.901 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:39.899+0000 7eff527fc640 10 monclient(hunting): _negotiate 2024-03-25T18:25:39.901 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:39.899+0000 7eff527fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:39.901 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:39.899+0000 7eff527fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:39.901 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:39.899+0000 7eff527fc640 10 monclient(hunting): my global_id is 4634 2024-03-25T18:25:39.901 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:39.899+0000 7eff527fc640 1 -- 172.21.15.97:0/1032344487 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff5c1d7960 con 0x7eff5c0b4d60 2024-03-25T18:25:39.903 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:39.902+0000 7eff527fc640 10 client.?.objecter ms_handle_connect 0x7eff5c14be90 2024-03-25T18:25:39.903 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:39.902+0000 7eff527fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:39.913 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:39.911+0000 7eff527fc640 1 -- 172.21.15.97:0/1032344487 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1326208483 0 0) 0x7eff5c1d8500 con 0x7eff5c14be90 2024-03-25T18:25:39.913 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:39.911+0000 7eff527fc640 10 monclient(hunting): _negotiate 2024-03-25T18:25:39.913 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:39.911+0000 7eff527fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:39.913 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:39.911+0000 7eff527fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:39.913 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:39.911+0000 7eff527fc640 10 monclient(hunting): my global_id is 4651 2024-03-25T18:25:39.913 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:39.911+0000 7eff527fc640 1 -- 172.21.15.97:0/1032344487 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff5c138850 con 0x7eff5c14be90 2024-03-25T18:25:39.914 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:39.912+0000 7eff527fc640 1 -- 172.21.15.97:0/1032344487 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1014843688 0 0) 0x7eff40004eb0 con 0x7eff5c0b1730 2024-03-25T18:25:39.914 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:39.912+0000 7eff527fc640 1 -- 172.21.15.97:0/1032344487 --> v1:172.21.15.97:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7eff5c1d8500 con 0x7eff5c0b1730 2024-03-25T18:25:39.914 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:39.912+0000 7eff527fc640 1 -- 172.21.15.97:0/1032344487 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7eff58001a10 con 0x7eff5c0b1730 2024-03-25T18:25:39.914 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:39.912+0000 7eff527fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:39.914 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:39.912+0000 7eff527fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:25:39.914 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:39.912+0000 7eff527fc640 10 monclient(hunting): dump: 2024-03-25T18:25:39.915 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:25:39.915 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:25:39.915 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:25:39.915 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:25:39.915 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:39.915 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:25:39.915 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:25:39.915 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:25:39.915 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:25:39.915 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:25:39.915 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:39.912+0000 7eff527fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:25:39.916 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:39.914+0000 7eff527fc640 1 -- 172.21.15.97:0/1032344487 <== mon.2 v1:172.21.15.97:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1668814412 0 0) 0x7eff5c1d7960 con 0x7eff5c0b4d60 2024-03-25T18:25:39.916 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:39.914+0000 7eff527fc640 1 -- 172.21.15.97:0/1032344487 --> v1:172.21.15.97:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7eff40004eb0 con 0x7eff5c0b4d60 2024-03-25T18:25:39.916 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:39.914+0000 7eff527fc640 1 -- 172.21.15.97:0/1032344487 <== mon.2 v1:172.21.15.97:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7eff4c001970 con 0x7eff5c0b4d60 2024-03-25T18:25:39.916 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:39.914+0000 7eff527fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:39.916 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:39.914+0000 7eff527fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:25:39.916 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:39.914+0000 7eff527fc640 10 monclient(hunting): dump: 2024-03-25T18:25:39.916 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:25:39.916 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:25:39.916 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:25:39.917 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:25:39.917 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:39.917 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:25:39.917 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:25:39.917 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:25:39.917 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:25:39.917 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:25:39.925 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:39.923+0000 7eff527fc640 1 -- 172.21.15.97:0/1032344487 <== mon.0 v1:172.21.15.97:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3601425755 0 0) 0x7eff5c1d8500 con 0x7eff5c0b1730 2024-03-25T18:25:39.925 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:39.923+0000 7eff527fc640 1 -- 172.21.15.97:0/1032344487 >> v1:172.21.15.97:6790/0 conn(0x7eff5c0b4d60 legacy=0x7eff5c1d7df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:39.925 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:39.923+0000 7eff527fc640 1 -- 172.21.15.97:0/1032344487 >> v1:172.21.15.98:6789/0 conn(0x7eff5c14be90 legacy=0x7eff5c142a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:39.925 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:39.923+0000 7eff527fc640 10 monclient: _finish_hunting 0 2024-03-25T18:25:39.925 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:39.923+0000 7eff527fc640 1 monclient: found mon.a 2024-03-25T18:25:39.925 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:39.923+0000 7eff527fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:39.925 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:39.923+0000 7eff527fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:25:39.925 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:39.923+0000 7eff527fc640 1 -- 172.21.15.97:0/1032344487 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff5c1389e0 con 0x7eff5c0b1730 2024-03-25T18:25:39.925 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:39.923+0000 7eff527fc640 10 monclient: _finish_auth 0 2024-03-25T18:25:39.925 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:39.923+0000 7eff527fc640 10 monclient: _check_auth_tickets 2024-03-25T18:25:39.925 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:39.923+0000 7eff527fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:39.925 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:39.923+0000 7eff63fd0640 5 monclient: authenticate success, global_id 4647 2024-03-25T18:25:39.925 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:39.923+0000 7eff63fd0640 10 monclient: _renew_subs 2024-03-25T18:25:39.926 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:39.923+0000 7eff63fd0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:25:39.926 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:39.923+0000 7eff63fd0640 1 -- 172.21.15.97:0/1032344487 --> v1:172.21.15.97:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff5c137e20 con 0x7eff5c0b1730 2024-03-25T18:25:39.926 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:39.923+0000 7eff63fd0640 10 client.4647.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:25:39.926 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:39.923+0000 7eff63fd0640 10 monclient: _renew_subs 2024-03-25T18:25:39.926 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:39.923+0000 7eff63fd0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:25:39.926 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:39.923+0000 7eff63fd0640 1 -- 172.21.15.97:0/1032344487 --> v1:172.21.15.97:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7eff5c143350 con 0x7eff5c0b1730 2024-03-25T18:25:39.926 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:39.923+0000 7eff63fd0640 1 librados: init done 2024-03-25T18:25:39.927 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:39.924+0000 7eff63fd0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:39.927 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:39.924+0000 7eff63fd0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:39.927 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:39.924+0000 7eff63fd0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:25:39.927 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:39.924+0000 7eff63fd0640 1 -- 172.21.15.97:0/1032344487 --> v1:172.21.15.97:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff5c0627b0 con 0x7eff5c0b1730 2024-03-25T18:25:40.232 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:40.230+0000 7eff527fc640 1 -- 172.21.15.97:0/1032344487 <== mon.0 v1:172.21.15.97:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7eff58001c70 con 0x7eff5c0b1730 2024-03-25T18:25:40.232 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:40.230+0000 7eff527fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:40.232 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:40.230+0000 7eff527fc640 1 -- 172.21.15.97:0/1032344487 <== mon.0 v1:172.21.15.97:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7eff58003a60 con 0x7eff5c0b1730 2024-03-25T18:25:40.232 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:40.230+0000 7eff527fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:40.232 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:40.230+0000 7eff527fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:25:40.232 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:40.230+0000 7eff527fc640 10 monclient: dump: 2024-03-25T18:25:40.232 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:25:40.232 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:25:40.232 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:25:40.232 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:25:40.233 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:40.233 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:25:40.233 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:25:40.233 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:25:40.233 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:25:40.233 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:25:40.233 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:40.230+0000 7eff53fff640 4 set_mon_vals no callback set 2024-03-25T18:25:40.234 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:40.230+0000 7eff527fc640 1 -- 172.21.15.97:0/1032344487 <== mon.0 v1:172.21.15.97:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2377622137 0 0) 0x7eff58003c80 con 0x7eff5c0b1730 2024-03-25T18:25:40.234 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:40.231+0000 7eff6156b640 10 monclient: get_auth_request con 0x7eff4006df70 auth_method 0 2024-03-25T18:25:40.234 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:40.231+0000 7eff527fc640 10 client.4647.objecter ms_handle_connect 0x7eff4006df70 2024-03-25T18:25:40.239 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:40.237+0000 7eff527fc640 1 -- 172.21.15.97:0/1032344487 <== mon.0 v1:172.21.15.97:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 430711918 0 0) 0x7eff58054820 con 0x7eff5c0b1730 2024-03-25T18:25:40.239 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:40.237+0000 7eff527fc640 10 client.4647.objecter ms_dispatch 0x7eff5c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:25:40.239 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:40.237+0000 7eff527fc640 3 client.4647.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:25:40.239 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:40.237+0000 7eff527fc640 3 client.4647.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:25:40.239 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:40.237+0000 7eff527fc640 20 client.4647.objecter dump_active .. 0 homeless 2024-03-25T18:25:40.239 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:40.237+0000 7eff527fc640 1 -- 172.21.15.97:0/1032344487 <== mon.0 v1:172.21.15.97:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7eff58050330 con 0x7eff5c0b1730 2024-03-25T18:25:40.239 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:40.237+0000 7eff527fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:40.239 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:40.237+0000 7eff527fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:25:40.480 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:40.478+0000 7eff63fd0640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:40.480 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:40.478+0000 7eff63fd0640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:40.480 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:40.478+0000 7eff63fd0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:25:40.480 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:40.478+0000 7eff63fd0640 1 -- 172.21.15.97:0/1032344487 --> v1:172.21.15.97:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7eff5c069f40 con 0x7eff5c0b1730 2024-03-25T18:25:40.501 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:40.500+0000 7eff527fc640 1 -- 172.21.15.97:0/1032344487 <== mon.0 v1:172.21.15.97:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2169427412) 0x7eff58052260 con 0x7eff5c0b1730 2024-03-25T18:25:40.501 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:40.500+0000 7eff527fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:40.501 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:40.500+0000 7eff527fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:25:40.502 INFO:teuthology.misc.health.smithi097.stdout: 2024-03-25T18:25:40.502 INFO:teuthology.misc.health.smithi097.stdout:{"epoch":10,"fsid":"4c3d472c-79b8-486b-9e3d-2738d4306754","created":"2024-03-25T18:16:14.737309+0000","modified":"2024-03-25T18:16:20.357492+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:20.357492+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cb5152b9-262f-45c3-94b8-18d0e81df558","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6daf8793-0838-473d-a7d6-9a0d2894c75d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"136f1493-3b9c-4f43-a2e7-641482d69dae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"397d538c-4f64-4daf-9e8b-662d09f91327","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4f1df08-663f-4bda-93f8-3878ab7c24b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c4b18d70-a3ce-4b15-9b70-eae5525bd761","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8dd4033a-de0e-4f8a-96dc-59f3c1f825b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0077722e-ffa6-4a3d-9891-929d9869b286","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:25:40.504 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:40.502+0000 7eff63fd0640 10 librados: watch_flush enter 2024-03-25T18:25:40.504 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:40.502+0000 7eff63fd0640 10 librados: watch_flush exit 2024-03-25T18:25:40.504 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:40.502+0000 7eff63fd0640 20 client.4647.objecter shutdown clearing up homeless session... 2024-03-25T18:25:40.504 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:40.502+0000 7eff63fd0640 10 client.4647.objecter successfully canceled tick 2024-03-25T18:25:40.504 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:40.502+0000 7eff63fd0640 1 -- 172.21.15.97:0/1032344487 >> v1:172.21.15.98:6800/800387573 conn(0x7eff4006df70 legacy=0x7eff40070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:40.504 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:40.502+0000 7eff63fd0640 10 monclient: shutdown 2024-03-25T18:25:40.505 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:40.502+0000 7eff63fd0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:40.505 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:40.502+0000 7eff63fd0640 1 -- 172.21.15.97:0/1032344487 >> v1:172.21.15.97:6789/0 conn(0x7eff5c0b1730 legacy=0x7eff5c1405f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:40.505 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:40.502+0000 7eff63fd0640 1 -- 172.21.15.97:0/1032344487 shutdown_connections 2024-03-25T18:25:40.505 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:40.502+0000 7eff63fd0640 1 -- 172.21.15.97:0/1032344487 >> 172.21.15.97:0/1032344487 conn(0x7eff5c0617a0 msgr2=0x7eff5c13c610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:40.505 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:40.502+0000 7eff63fd0640 1 -- 172.21.15.97:0/1032344487 shutdown_connections 2024-03-25T18:25:40.505 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:40.502+0000 7eff63fd0640 1 -- 172.21.15.97:0/1032344487 wait complete. 2024-03-25T18:25:40.505 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:40.502+0000 7eff63fd0640 1 librados: shutdown 2024-03-25T18:25:40.514 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:25:47.321 DEBUG:teuthology.orchestra.run.smithi097:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:25:47.440 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.438+0000 7facc600d640 10 monclient: get_monmap_and_config 2024-03-25T18:25:47.440 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.438+0000 7facc600d640 10 monclient: build_initial_monmap 2024-03-25T18:25:47.440 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.438+0000 7facc600d640 10 monclient: monmap: 2024-03-25T18:25:47.440 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:25:47.440 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:25:47.440 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:25:47.439848+0000 2024-03-25T18:25:47.440 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:25:47.439848+0000 2024-03-25T18:25:47.440 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:47.441 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:25:47.441 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:25:47.441 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T18:25:47.441 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-b 2024-03-25T18:25:47.441 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:25:47.442 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.439+0000 7facc600d640 1 Processor -- start 2024-03-25T18:25:47.442 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.439+0000 7facc600d640 1 -- start start 2024-03-25T18:25:47.442 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.439+0000 7facc600d640 10 monclient: init 2024-03-25T18:25:47.442 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.440+0000 7facc600d640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:47.442 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.440+0000 7facc600d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:25:47.442 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.440+0000 7facc600d640 10 monclient(hunting): picked mon.noname-a con 0x7facc00a98f0 addr v1:172.21.15.97:6789/0 2024-03-25T18:25:47.442 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.440+0000 7facc600d640 10 monclient(hunting): picked mon.noname-b con 0x7facc013f710 addr v1:172.21.15.97:6790/0 2024-03-25T18:25:47.442 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.440+0000 7facc600d640 10 monclient(hunting): picked mon.noname-c con 0x7facc01430c0 addr v1:172.21.15.98:6789/0 2024-03-25T18:25:47.443 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.440+0000 7facc600d640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facc0142520 con 0x7facc00a98f0 2024-03-25T18:25:47.443 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.440+0000 7facc600d640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facc00b28a0 con 0x7facc01430c0 2024-03-25T18:25:47.443 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.440+0000 7facc600d640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facc00b2a30 con 0x7facc013f710 2024-03-25T18:25:47.443 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.440+0000 7facc600d640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:47.443 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.440+0000 7facc600d640 10 monclient(hunting): authenticate will time out at 8159.716103s 2024-03-25T18:25:47.455 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.454+0000 7facbf7fe640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7facc00a98f0 0x7facc00a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:41470/0 (socket says 172.21.15.97:41470) 2024-03-25T18:25:47.455 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.454+0000 7facbf7fe640 1 -- 172.21.15.97:0/2514726497 learned_addr learned my addr 172.21.15.97:0/2514726497 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:25:47.467 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.465+0000 7facbe7fc640 1 -- 172.21.15.97:0/2514726497 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3257069991 0 0) 0x7facc00b28a0 con 0x7facc01430c0 2024-03-25T18:25:47.467 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.465+0000 7facbe7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:25:47.467 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.465+0000 7facbe7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:47.467 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.465+0000 7facbe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:47.468 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.465+0000 7facbe7fc640 10 monclient(hunting): my global_id is 4654 2024-03-25T18:25:47.468 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.465+0000 7facbe7fc640 1 -- 172.21.15.97:0/2514726497 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faca8003520 con 0x7facc01430c0 2024-03-25T18:25:47.477 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.475+0000 7facbe7fc640 1 -- 172.21.15.97:0/2514726497 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 573569500 0 0) 0x7facc0142520 con 0x7facc00a98f0 2024-03-25T18:25:47.477 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.475+0000 7facbe7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:25:47.477 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.475+0000 7facbe7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:47.477 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.475+0000 7facbe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:47.477 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.475+0000 7facbe7fc640 10 monclient(hunting): my global_id is 4650 2024-03-25T18:25:47.477 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.475+0000 7facbe7fc640 1 -- 172.21.15.97:0/2514726497 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7facc00b28a0 con 0x7facc00a98f0 2024-03-25T18:25:47.483 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.481+0000 7facbe7fc640 1 -- 172.21.15.97:0/2514726497 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1175019787 0 0) 0x7facc00b28a0 con 0x7facc00a98f0 2024-03-25T18:25:47.483 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.481+0000 7facbe7fc640 1 -- 172.21.15.97:0/2514726497 >> v1:172.21.15.97:6790/0 conn(0x7facc013f710 legacy=0x7facc01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:47.483 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.481+0000 7facbe7fc640 1 -- 172.21.15.97:0/2514726497 >> v1:172.21.15.98:6789/0 conn(0x7facc01430c0 legacy=0x7facc014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:47.483 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.481+0000 7facbe7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:25:47.483 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.481+0000 7facbe7fc640 1 monclient: found mon.noname-a 2024-03-25T18:25:47.483 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.481+0000 7facbe7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:47.484 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.481+0000 7facbe7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.97:6789/0 2024-03-25T18:25:47.484 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.481+0000 7facbe7fc640 1 -- 172.21.15.97:0/2514726497 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facc00b44c0 con 0x7facc00a98f0 2024-03-25T18:25:47.484 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.481+0000 7facbe7fc640 10 monclient: _finish_auth 0 2024-03-25T18:25:47.484 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.481+0000 7facbe7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:25:47.484 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.481+0000 7facbe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:47.484 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.481+0000 7facbe7fc640 1 -- 172.21.15.97:0/2514726497 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7facb0001670 con 0x7facc00a98f0 2024-03-25T18:25:47.484 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.481+0000 7facbe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:47.484 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.481+0000 7facbe7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:25:47.484 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.481+0000 7facbe7fc640 10 monclient: dump: 2024-03-25T18:25:47.484 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:25:47.484 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:25:47.484 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:25:47.484 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:25:47.484 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:47.485 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:25:47.485 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:25:47.485 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:25:47.485 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:25:47.485 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:25:47.485 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.481+0000 7facc600d640 5 monclient: authenticate success, global_id 4650 2024-03-25T18:25:47.485 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.481+0000 7facc600d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:25:47.486 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.483+0000 7facbe7fc640 1 -- 172.21.15.97:0/2514726497 <== mon.0 v1:172.21.15.97:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7facb0001850 con 0x7facc00a98f0 2024-03-25T18:25:47.486 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.484+0000 7facbe7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:47.486 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.484+0000 7facc600d640 10 monclient: get_monmap_and_config success 2024-03-25T18:25:47.487 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.484+0000 7facc600d640 4 set_mon_vals no callback set 2024-03-25T18:25:47.487 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.484+0000 7facbe7fc640 1 -- 172.21.15.97:0/2514726497 <== mon.0 v1:172.21.15.97:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7facb0001be0 con 0x7facc00a98f0 2024-03-25T18:25:47.487 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.484+0000 7facbe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:47.487 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.484+0000 7facbe7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:25:47.487 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.484+0000 7facbe7fc640 10 monclient: dump: 2024-03-25T18:25:47.487 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:25:47.487 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:25:47.487 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:25:47.487 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:25:47.487 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:47.487 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:25:47.487 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:25:47.487 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:25:47.488 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:25:47.488 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:25:47.488 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.484+0000 7facc600d640 10 monclient: shutdown 2024-03-25T18:25:47.488 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.484+0000 7facc600d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:47.488 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.484+0000 7facc600d640 1 -- 172.21.15.97:0/2514726497 >> v1:172.21.15.97:6789/0 conn(0x7facc00a98f0 legacy=0x7facc00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:47.488 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.484+0000 7facc600d640 1 -- 172.21.15.97:0/2514726497 shutdown_connections 2024-03-25T18:25:47.488 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.484+0000 7facc600d640 1 -- 172.21.15.97:0/2514726497 >> 172.21.15.97:0/2514726497 conn(0x7facc00b7ea0 msgr2=0x7facc00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:47.488 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.484+0000 7facc600d640 1 -- 172.21.15.97:0/2514726497 shutdown_connections 2024-03-25T18:25:47.488 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.484+0000 7facc600d640 1 -- 172.21.15.97:0/2514726497 wait complete. 2024-03-25T18:25:47.490 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.485+0000 7facc600d640 10 monclient: build_initial_monmap 2024-03-25T18:25:47.490 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.485+0000 7facc600d640 10 monclient: monmap: 2024-03-25T18:25:47.490 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:25:47.490 INFO:teuthology.misc.health.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:25:47.490 INFO:teuthology.misc.health.smithi097.stderr:last_changed 0.000000 2024-03-25T18:25:47.490 INFO:teuthology.misc.health.smithi097.stderr:created 0.000000 2024-03-25T18:25:47.490 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:47.490 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 1 2024-03-25T18:25:47.490 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:25:47.490 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T18:25:47.491 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-c 2024-03-25T18:25:47.491 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:25:47.491 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.486+0000 7facc600d640 1 librados: starting msgr at 2024-03-25T18:25:47.491 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.486+0000 7facc600d640 1 librados: starting objecter 2024-03-25T18:25:47.491 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.486+0000 7facc600d640 1 Processor -- start 2024-03-25T18:25:47.491 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.486+0000 7facc600d640 1 -- start start 2024-03-25T18:25:47.491 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.486+0000 7facc600d640 1 librados: setting wanted keys 2024-03-25T18:25:47.491 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.486+0000 7facc600d640 1 librados: calling monclient init 2024-03-25T18:25:47.491 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.486+0000 7facc600d640 10 monclient: init 2024-03-25T18:25:47.491 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.486+0000 7facc600d640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:47.491 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.486+0000 7facc600d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:25:47.491 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.486+0000 7facc600d640 10 monclient(hunting): picked mon.noname-b con 0x7facc00a98f0 addr v1:172.21.15.98:6789/0 2024-03-25T18:25:47.491 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.486+0000 7facc600d640 10 monclient(hunting): picked mon.noname-a con 0x7facc013f710 addr v1:172.21.15.97:6789/0 2024-03-25T18:25:47.491 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.486+0000 7facc600d640 10 monclient(hunting): picked mon.noname-c con 0x7facc01430c0 addr v1:172.21.15.97:6790/0 2024-03-25T18:25:47.491 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.486+0000 7facc600d640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facc01dee90 con 0x7facc013f710 2024-03-25T18:25:47.492 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.486+0000 7facc600d640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facc01dfac0 con 0x7facc00a98f0 2024-03-25T18:25:47.492 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.486+0000 7facc600d640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facc01dfc50 con 0x7facc01430c0 2024-03-25T18:25:47.492 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.486+0000 7facc600d640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:47.492 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.486+0000 7facc600d640 10 monclient(hunting): authenticate will time out at 8159.762810s 2024-03-25T18:25:47.501 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.499+0000 7facbf7fe640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7facc00a98f0 0x7facc01df3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.97:56100/0 (socket says 172.21.15.97:56100) 2024-03-25T18:25:47.501 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.499+0000 7facbf7fe640 1 -- 172.21.15.97:0/3144861666 learned_addr learned my addr 172.21.15.97:0/3144861666 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:25:47.502 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.500+0000 7faca7fff640 10 client.?.objecter ms_handle_connect 0x7facc00a98f0 2024-03-25T18:25:47.502 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.500+0000 7faca7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:47.504 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.503+0000 7faca7fff640 10 client.?.objecter ms_handle_connect 0x7facc013f710 2024-03-25T18:25:47.504 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.503+0000 7faca7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:47.511 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.510+0000 7faca7fff640 1 -- 172.21.15.97:0/3144861666 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3374673725 0 0) 0x7facc01dfac0 con 0x7facc00a98f0 2024-03-25T18:25:47.511 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.510+0000 7faca7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:25:47.511 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.510+0000 7faca7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:47.511 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.510+0000 7faca7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:47.511 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.510+0000 7faca7fff640 10 monclient(hunting): my global_id is 4657 2024-03-25T18:25:47.511 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.510+0000 7faca7fff640 1 -- 172.21.15.97:0/3144861666 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7facac004eb0 con 0x7facc00a98f0 2024-03-25T18:25:47.515 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.514+0000 7faca7fff640 1 -- 172.21.15.97:0/3144861666 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3437382745 0 0) 0x7facc01dee90 con 0x7facc013f710 2024-03-25T18:25:47.515 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.514+0000 7faca7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:25:47.515 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.514+0000 7faca7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:47.515 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.514+0000 7faca7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:47.515 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.514+0000 7faca7fff640 10 monclient(hunting): my global_id is 4653 2024-03-25T18:25:47.516 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.514+0000 7faca7fff640 1 -- 172.21.15.97:0/3144861666 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7facc01dfac0 con 0x7facc013f710 2024-03-25T18:25:47.522 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.520+0000 7faca7fff640 10 client.?.objecter ms_handle_connect 0x7facc01430c0 2024-03-25T18:25:47.522 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.520+0000 7faca7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:47.526 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.524+0000 7faca7fff640 1 -- 172.21.15.97:0/3144861666 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2115853231 0 0) 0x7facac004eb0 con 0x7facc00a98f0 2024-03-25T18:25:47.526 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.524+0000 7faca7fff640 1 -- 172.21.15.97:0/3144861666 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7facc01dee90 con 0x7facc00a98f0 2024-03-25T18:25:47.526 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.524+0000 7faca7fff640 1 -- 172.21.15.97:0/3144861666 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7facb0001c20 con 0x7facc00a98f0 2024-03-25T18:25:47.526 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.524+0000 7faca7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:47.526 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.524+0000 7faca7fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:25:47.526 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.524+0000 7faca7fff640 10 monclient(hunting): dump: 2024-03-25T18:25:47.526 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:25:47.526 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:25:47.526 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:25:47.527 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:25:47.527 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:47.527 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:25:47.527 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:25:47.527 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:25:47.527 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:25:47.527 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:25:47.527 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.524+0000 7faca7fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:25:47.532 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.531+0000 7faca7fff640 1 -- 172.21.15.97:0/3144861666 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3068902628 0 0) 0x7facc01dfac0 con 0x7facc013f710 2024-03-25T18:25:47.533 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.531+0000 7faca7fff640 1 -- 172.21.15.97:0/3144861666 --> v1:172.21.15.97:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7facac004eb0 con 0x7facc013f710 2024-03-25T18:25:47.533 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.531+0000 7faca7fff640 1 -- 172.21.15.97:0/3144861666 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7facb40018f0 con 0x7facc013f710 2024-03-25T18:25:47.533 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.531+0000 7faca7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:47.534 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.531+0000 7faca7fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:25:47.534 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.531+0000 7faca7fff640 10 monclient(hunting): dump: 2024-03-25T18:25:47.534 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:25:47.534 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:25:47.534 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:25:47.534 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:25:47.534 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:47.534 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:25:47.534 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:25:47.534 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:25:47.534 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:25:47.534 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:25:47.535 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.532+0000 7faca7fff640 1 -- 172.21.15.97:0/3144861666 <== mon.1 v1:172.21.15.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2594303401 0 0) 0x7facc01dee90 con 0x7facc00a98f0 2024-03-25T18:25:47.536 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.532+0000 7faca7fff640 1 -- 172.21.15.97:0/3144861666 >> v1:172.21.15.97:6790/0 conn(0x7facc01430c0 legacy=0x7facc01e6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:47.536 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.532+0000 7faca7fff640 1 -- 172.21.15.97:0/3144861666 >> v1:172.21.15.97:6789/0 conn(0x7facc013f710 legacy=0x7facc006e080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:47.536 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.532+0000 7faca7fff640 10 monclient: _finish_hunting 0 2024-03-25T18:25:47.536 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.532+0000 7faca7fff640 1 monclient: found mon.b 2024-03-25T18:25:47.536 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.532+0000 7faca7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:47.536 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.532+0000 7faca7fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:25:47.536 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.532+0000 7faca7fff640 1 -- 172.21.15.97:0/3144861666 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facc006e790 con 0x7facc00a98f0 2024-03-25T18:25:47.536 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.532+0000 7faca7fff640 10 monclient: _finish_auth 0 2024-03-25T18:25:47.536 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.532+0000 7faca7fff640 10 monclient: _check_auth_tickets 2024-03-25T18:25:47.537 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.532+0000 7faca7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:47.537 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.532+0000 7facc600d640 5 monclient: authenticate success, global_id 4657 2024-03-25T18:25:47.537 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.532+0000 7facc600d640 10 monclient: _renew_subs 2024-03-25T18:25:47.537 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.532+0000 7facc600d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:25:47.537 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.532+0000 7facc600d640 1 -- 172.21.15.97:0/3144861666 --> v1:172.21.15.98:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facc006f0a0 con 0x7facc00a98f0 2024-03-25T18:25:47.537 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.532+0000 7facc600d640 10 client.4657.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:25:47.537 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.532+0000 7facc600d640 10 monclient: _renew_subs 2024-03-25T18:25:47.537 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.532+0000 7facc600d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:25:47.537 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.532+0000 7facc600d640 1 -- 172.21.15.97:0/3144861666 --> v1:172.21.15.98:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7facc0070870 con 0x7facc00a98f0 2024-03-25T18:25:47.537 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.532+0000 7facc600d640 1 librados: init done 2024-03-25T18:25:47.537 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.533+0000 7facc600d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:47.537 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.533+0000 7facc600d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:47.537 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.533+0000 7facc600d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:25:47.537 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.533+0000 7facc600d640 1 -- 172.21.15.97:0/3144861666 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facc014f320 con 0x7facc00a98f0 2024-03-25T18:25:47.746 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.744+0000 7faca7fff640 1 -- 172.21.15.97:0/3144861666 <== mon.1 v1:172.21.15.98:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7facb0001220 con 0x7facc00a98f0 2024-03-25T18:25:47.746 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.744+0000 7faca7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:47.746 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.744+0000 7facbdffb640 4 set_mon_vals no callback set 2024-03-25T18:25:47.801 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.799+0000 7faca7fff640 1 -- 172.21.15.97:0/3144861666 <== mon.1 v1:172.21.15.98:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7facb0004cf0 con 0x7facc00a98f0 2024-03-25T18:25:47.801 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.799+0000 7faca7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:47.801 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.799+0000 7faca7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:25:47.801 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.799+0000 7faca7fff640 10 monclient: dump: 2024-03-25T18:25:47.801 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:25:47.801 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:25:47.801 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:25:47.801 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:25:47.801 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:47.801 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:25:47.801 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:25:47.801 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:25:47.801 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:25:47.802 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:25:47.802 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.800+0000 7faca7fff640 1 -- 172.21.15.97:0/3144861666 <== mon.1 v1:172.21.15.98:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2377622137 0 0) 0x7facb0004ed0 con 0x7facc00a98f0 2024-03-25T18:25:47.803 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.801+0000 7facbeffd640 10 monclient: get_auth_request con 0x7facac06de40 auth_method 0 2024-03-25T18:25:47.803 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.801+0000 7faca7fff640 10 client.4657.objecter ms_handle_connect 0x7facac06de40 2024-03-25T18:25:47.804 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.802+0000 7faca7fff640 1 -- 172.21.15.97:0/3144861666 <== mon.1 v1:172.21.15.98:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 430711918 0 0) 0x7facb0054ad0 con 0x7facc00a98f0 2024-03-25T18:25:47.804 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.802+0000 7faca7fff640 10 client.4657.objecter ms_dispatch 0x7facc0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:25:47.804 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.802+0000 7faca7fff640 3 client.4657.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:25:47.804 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.802+0000 7faca7fff640 3 client.4657.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:25:47.804 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.803+0000 7faca7fff640 20 client.4657.objecter dump_active .. 0 homeless 2024-03-25T18:25:47.808 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.806+0000 7faca7fff640 1 -- 172.21.15.97:0/3144861666 <== mon.1 v1:172.21.15.98:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7facb0050410 con 0x7facc00a98f0 2024-03-25T18:25:47.808 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.806+0000 7faca7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:47.808 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:47.806+0000 7faca7fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:25:48.060 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:48.058+0000 7facc600d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:48.060 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:48.058+0000 7facc600d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:48.060 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:48.058+0000 7facc600d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:25:48.060 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:48.058+0000 7facc600d640 1 -- 172.21.15.97:0/3144861666 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7facc0070bc0 con 0x7facc00a98f0 2024-03-25T18:25:48.077 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:48.075+0000 7faca7fff640 1 -- 172.21.15.97:0/3144861666 <== mon.1 v1:172.21.15.98:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2169427412) 0x7facb0052510 con 0x7facc00a98f0 2024-03-25T18:25:48.077 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:48.076+0000 7faca7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:48.077 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:48.076+0000 7faca7fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:25:48.077 INFO:teuthology.misc.health.smithi097.stdout: 2024-03-25T18:25:48.078 INFO:teuthology.misc.health.smithi097.stdout:{"epoch":10,"fsid":"4c3d472c-79b8-486b-9e3d-2738d4306754","created":"2024-03-25T18:16:14.737309+0000","modified":"2024-03-25T18:16:20.357492+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:20.357492+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cb5152b9-262f-45c3-94b8-18d0e81df558","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6daf8793-0838-473d-a7d6-9a0d2894c75d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"136f1493-3b9c-4f43-a2e7-641482d69dae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"397d538c-4f64-4daf-9e8b-662d09f91327","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4f1df08-663f-4bda-93f8-3878ab7c24b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c4b18d70-a3ce-4b15-9b70-eae5525bd761","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8dd4033a-de0e-4f8a-96dc-59f3c1f825b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0077722e-ffa6-4a3d-9891-929d9869b286","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:25:48.080 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:48.078+0000 7facc600d640 10 librados: watch_flush enter 2024-03-25T18:25:48.080 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:48.078+0000 7facc600d640 10 librados: watch_flush exit 2024-03-25T18:25:48.080 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:48.078+0000 7facc600d640 20 client.4657.objecter shutdown clearing up homeless session... 2024-03-25T18:25:48.080 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:48.078+0000 7facc600d640 10 client.4657.objecter successfully canceled tick 2024-03-25T18:25:48.080 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:48.078+0000 7facc600d640 1 -- 172.21.15.97:0/3144861666 >> v1:172.21.15.98:6800/800387573 conn(0x7facac06de40 legacy=0x7facac070300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:48.080 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:48.078+0000 7facc600d640 10 monclient: shutdown 2024-03-25T18:25:48.080 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:48.078+0000 7facc600d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:48.080 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:48.078+0000 7facc600d640 1 -- 172.21.15.97:0/3144861666 >> v1:172.21.15.98:6789/0 conn(0x7facc00a98f0 legacy=0x7facc01df3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:48.081 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:48.078+0000 7facc600d640 1 -- 172.21.15.97:0/3144861666 shutdown_connections 2024-03-25T18:25:48.081 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:48.078+0000 7facc600d640 1 -- 172.21.15.97:0/3144861666 >> 172.21.15.97:0/3144861666 conn(0x7facc00b7ea0 msgr2=0x7facc0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:48.081 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:48.078+0000 7facc600d640 1 -- 172.21.15.97:0/3144861666 shutdown_connections 2024-03-25T18:25:48.081 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:48.078+0000 7facc600d640 1 -- 172.21.15.97:0/3144861666 wait complete. 2024-03-25T18:25:48.081 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:48.078+0000 7facc600d640 1 librados: shutdown 2024-03-25T18:25:48.091 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:25:54.898 DEBUG:teuthology.orchestra.run.smithi097:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:25:55.015 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.014+0000 7f95082a8640 10 monclient: get_monmap_and_config 2024-03-25T18:25:55.016 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.014+0000 7f95082a8640 10 monclient: build_initial_monmap 2024-03-25T18:25:55.016 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.014+0000 7f95082a8640 10 monclient: monmap: 2024-03-25T18:25:55.016 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:25:55.016 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:25:55.016 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:25:55.015439+0000 2024-03-25T18:25:55.016 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:25:55.015439+0000 2024-03-25T18:25:55.016 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:55.016 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:25:55.016 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:25:55.016 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T18:25:55.016 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-b 2024-03-25T18:25:55.017 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:25:55.018 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.014+0000 7f95082a8640 1 Processor -- start 2024-03-25T18:25:55.018 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.014+0000 7f95082a8640 1 -- start start 2024-03-25T18:25:55.018 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.014+0000 7f95082a8640 10 monclient: init 2024-03-25T18:25:55.018 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.015+0000 7f95082a8640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:55.018 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.015+0000 7f95082a8640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:25:55.018 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.015+0000 7f95082a8640 10 monclient(hunting): picked mon.noname-b con 0x7f95000a98f0 addr v1:172.21.15.97:6790/0 2024-03-25T18:25:55.018 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.015+0000 7f95082a8640 10 monclient(hunting): picked mon.noname-a con 0x7f950013f710 addr v1:172.21.15.97:6789/0 2024-03-25T18:25:55.019 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.016+0000 7f95082a8640 10 monclient(hunting): picked mon.noname-c con 0x7f95001430c0 addr v1:172.21.15.98:6789/0 2024-03-25T18:25:55.019 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.016+0000 7f95082a8640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9500142520 con 0x7f950013f710 2024-03-25T18:25:55.019 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.016+0000 7f95082a8640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95000b28a0 con 0x7f95001430c0 2024-03-25T18:25:55.019 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.016+0000 7f95082a8640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95000b2a30 con 0x7f95000a98f0 2024-03-25T18:25:55.019 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.016+0000 7f95082a8640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:55.019 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.016+0000 7f95082a8640 10 monclient(hunting): authenticate will time out at 8167.291978s 2024-03-25T18:25:55.031 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.029+0000 7f9506845640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f95001430c0 0x7f950014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.97:49212/0 (socket says 172.21.15.97:49212) 2024-03-25T18:25:55.031 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.029+0000 7f9506845640 1 -- 172.21.15.97:0/309304263 learned_addr learned my addr 172.21.15.97:0/309304263 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:25:55.045 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.043+0000 7f9505042640 1 -- 172.21.15.97:0/309304263 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1927069696 0 0) 0x7f95000b28a0 con 0x7f95001430c0 2024-03-25T18:25:55.045 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.043+0000 7f9505042640 10 monclient(hunting): _negotiate 2024-03-25T18:25:55.045 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.043+0000 7f9505042640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:55.045 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.043+0000 7f9505042640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:55.045 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.043+0000 7f9505042640 10 monclient(hunting): my global_id is 4660 2024-03-25T18:25:55.045 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.043+0000 7f9505042640 1 -- 172.21.15.97:0/309304263 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94ec003520 con 0x7f95001430c0 2024-03-25T18:25:55.054 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.052+0000 7f9505042640 1 -- 172.21.15.97:0/309304263 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 454850522 0 0) 0x7f9500142520 con 0x7f950013f710 2024-03-25T18:25:55.054 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.052+0000 7f9505042640 10 monclient(hunting): _negotiate 2024-03-25T18:25:55.054 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.052+0000 7f9505042640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:55.054 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.052+0000 7f9505042640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:55.054 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.052+0000 7f9505042640 10 monclient(hunting): my global_id is 4656 2024-03-25T18:25:55.054 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.052+0000 7f9505042640 1 -- 172.21.15.97:0/309304263 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95000b28a0 con 0x7f950013f710 2024-03-25T18:25:55.055 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.053+0000 7f9505042640 1 -- 172.21.15.97:0/309304263 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2603986617 0 0) 0x7f95000b2a30 con 0x7f95000a98f0 2024-03-25T18:25:55.055 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.053+0000 7f9505042640 10 monclient(hunting): _negotiate 2024-03-25T18:25:55.055 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.053+0000 7f9505042640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:55.055 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.053+0000 7f9505042640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:55.055 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.053+0000 7f9505042640 10 monclient(hunting): my global_id is 4640 2024-03-25T18:25:55.055 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.053+0000 7f9505042640 1 -- 172.21.15.97:0/309304263 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9500142520 con 0x7f95000a98f0 2024-03-25T18:25:55.062 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.061+0000 7f9505042640 1 -- 172.21.15.97:0/309304263 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3618406367 0 0) 0x7f94ec003520 con 0x7f95001430c0 2024-03-25T18:25:55.062 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.061+0000 7f9505042640 1 -- 172.21.15.97:0/309304263 >> v1:172.21.15.97:6790/0 conn(0x7f95000a98f0 legacy=0x7f95000a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:55.063 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.061+0000 7f9505042640 1 -- 172.21.15.97:0/309304263 >> v1:172.21.15.97:6789/0 conn(0x7f950013f710 legacy=0x7f95001429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:55.063 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.061+0000 7f9505042640 10 monclient: _finish_hunting 0 2024-03-25T18:25:55.063 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.061+0000 7f9505042640 1 monclient: found mon.noname-c 2024-03-25T18:25:55.063 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.061+0000 7f9505042640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:55.063 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.061+0000 7f9505042640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.98:6789/0 2024-03-25T18:25:55.063 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.061+0000 7f9505042640 1 -- 172.21.15.97:0/309304263 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95000b44c0 con 0x7f95001430c0 2024-03-25T18:25:55.063 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.061+0000 7f9505042640 10 monclient: _finish_auth 0 2024-03-25T18:25:55.063 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.061+0000 7f9505042640 10 monclient: _check_auth_tickets 2024-03-25T18:25:55.063 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.061+0000 7f9505042640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:55.063 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.061+0000 7f9505042640 1 -- 172.21.15.97:0/309304263 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f94f8001a10 con 0x7f95001430c0 2024-03-25T18:25:55.063 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.061+0000 7f95082a8640 5 monclient: authenticate success, global_id 4660 2024-03-25T18:25:55.063 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.061+0000 7f95082a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:25:55.064 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.061+0000 7f9505042640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:55.064 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.061+0000 7f9505042640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:25:55.064 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.061+0000 7f9505042640 10 monclient: dump: 2024-03-25T18:25:55.064 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:25:55.064 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:25:55.064 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:25:55.064 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:25:55.064 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:55.064 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:25:55.064 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:25:55.064 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:25:55.064 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:25:55.064 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:25:55.065 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.061+0000 7f95082a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:25:55.075 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.073+0000 7f9505042640 1 -- 172.21.15.97:0/309304263 <== mon.1 v1:172.21.15.98:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f94f8001b70 con 0x7f95001430c0 2024-03-25T18:25:55.075 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.073+0000 7f9505042640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:55.075 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.073+0000 7f9505042640 1 -- 172.21.15.97:0/309304263 <== mon.1 v1:172.21.15.98:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f94f8001f80 con 0x7f95001430c0 2024-03-25T18:25:55.075 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.073+0000 7f9505042640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:55.075 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.073+0000 7f9505042640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:25:55.076 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.073+0000 7f9505042640 10 monclient: dump: 2024-03-25T18:25:55.076 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:25:55.076 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:25:55.076 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:25:55.076 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:25:55.076 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:55.076 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:25:55.076 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:25:55.076 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:25:55.076 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:25:55.076 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:25:55.076 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.073+0000 7f95082a8640 10 monclient: get_monmap_and_config success 2024-03-25T18:25:55.077 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.073+0000 7f95082a8640 4 set_mon_vals no callback set 2024-03-25T18:25:55.079 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.074+0000 7f95082a8640 10 monclient: shutdown 2024-03-25T18:25:55.079 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.074+0000 7f95082a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:55.079 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.074+0000 7f95082a8640 1 -- 172.21.15.97:0/309304263 >> v1:172.21.15.98:6789/0 conn(0x7f95001430c0 legacy=0x7f950014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:55.079 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.074+0000 7f95082a8640 1 -- 172.21.15.97:0/309304263 shutdown_connections 2024-03-25T18:25:55.079 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.074+0000 7f95082a8640 1 -- 172.21.15.97:0/309304263 >> 172.21.15.97:0/309304263 conn(0x7f95000b7ea0 msgr2=0x7f95000b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:55.079 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.074+0000 7f95082a8640 1 -- 172.21.15.97:0/309304263 shutdown_connections 2024-03-25T18:25:55.079 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.074+0000 7f95082a8640 1 -- 172.21.15.97:0/309304263 wait complete. 2024-03-25T18:25:55.079 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.075+0000 7f95082a8640 10 monclient: build_initial_monmap 2024-03-25T18:25:55.079 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.075+0000 7f95082a8640 10 monclient: monmap: 2024-03-25T18:25:55.080 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:25:55.080 INFO:teuthology.misc.health.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:25:55.080 INFO:teuthology.misc.health.smithi097.stderr:last_changed 0.000000 2024-03-25T18:25:55.080 INFO:teuthology.misc.health.smithi097.stderr:created 0.000000 2024-03-25T18:25:55.080 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:55.080 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 1 2024-03-25T18:25:55.080 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:25:55.080 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T18:25:55.080 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-c 2024-03-25T18:25:55.080 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:25:55.080 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.075+0000 7f95082a8640 1 librados: starting msgr at 2024-03-25T18:25:55.080 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.075+0000 7f95082a8640 1 librados: starting objecter 2024-03-25T18:25:55.080 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.075+0000 7f95082a8640 1 Processor -- start 2024-03-25T18:25:55.081 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.075+0000 7f95082a8640 1 -- start start 2024-03-25T18:25:55.081 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.075+0000 7f95082a8640 1 librados: setting wanted keys 2024-03-25T18:25:55.081 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.075+0000 7f95082a8640 1 librados: calling monclient init 2024-03-25T18:25:55.081 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.075+0000 7f95082a8640 10 monclient: init 2024-03-25T18:25:55.081 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.076+0000 7f95082a8640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:55.081 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.076+0000 7f95082a8640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:25:55.081 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.076+0000 7f95082a8640 10 monclient(hunting): picked mon.noname-a con 0x7f95000a98f0 addr v1:172.21.15.97:6789/0 2024-03-25T18:25:55.081 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.076+0000 7f95082a8640 10 monclient(hunting): picked mon.noname-c con 0x7f950013f710 addr v1:172.21.15.97:6790/0 2024-03-25T18:25:55.081 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.076+0000 7f95082a8640 10 monclient(hunting): picked mon.noname-b con 0x7f95001430c0 addr v1:172.21.15.98:6789/0 2024-03-25T18:25:55.081 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.076+0000 7f95082a8640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95000ab920 con 0x7f95000a98f0 2024-03-25T18:25:55.081 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.076+0000 7f95082a8640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9500135140 con 0x7f95001430c0 2024-03-25T18:25:55.081 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.076+0000 7f95082a8640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9500134e10 con 0x7f950013f710 2024-03-25T18:25:55.081 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.076+0000 7f95082a8640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:55.081 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.076+0000 7f95082a8640 10 monclient(hunting): authenticate will time out at 8167.352270s 2024-03-25T18:25:55.087 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.085+0000 7f9506845640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f95001430c0 0x7f9500135660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.97:49232/0 (socket says 172.21.15.97:49232) 2024-03-25T18:25:55.087 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.085+0000 7f9506845640 1 -- 172.21.15.97:0/3221149025 learned_addr learned my addr 172.21.15.97:0/3221149025 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:25:55.088 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.086+0000 7f94f6ffd640 10 client.?.objecter ms_handle_connect 0x7f95001430c0 2024-03-25T18:25:55.088 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.086+0000 7f94f6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:55.093 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.091+0000 7f94f6ffd640 10 client.?.objecter ms_handle_connect 0x7f950013f710 2024-03-25T18:25:55.093 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.091+0000 7f94f6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:55.098 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.097+0000 7f94f6ffd640 10 client.?.objecter ms_handle_connect 0x7f95000a98f0 2024-03-25T18:25:55.098 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.097+0000 7f94f6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:55.101 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.099+0000 7f94f6ffd640 1 -- 172.21.15.97:0/3221149025 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3560102227 0 0) 0x7f9500135140 con 0x7f95001430c0 2024-03-25T18:25:55.101 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.100+0000 7f94f6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:25:55.101 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.100+0000 7f94f6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:55.102 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.100+0000 7f94f6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:55.102 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.100+0000 7f94f6ffd640 10 monclient(hunting): my global_id is 4663 2024-03-25T18:25:55.102 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.100+0000 7f94f6ffd640 1 -- 172.21.15.97:0/3221149025 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94e4004eb0 con 0x7f95001430c0 2024-03-25T18:25:55.102 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.100+0000 7f94f6ffd640 1 -- 172.21.15.97:0/3221149025 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 122848831 0 0) 0x7f9500134e10 con 0x7f950013f710 2024-03-25T18:25:55.102 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.100+0000 7f94f6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:25:55.102 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.100+0000 7f94f6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:55.102 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.100+0000 7f94f6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:55.102 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.100+0000 7f94f6ffd640 10 monclient(hunting): my global_id is 4643 2024-03-25T18:25:55.102 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.100+0000 7f94f6ffd640 1 -- 172.21.15.97:0/3221149025 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9500135140 con 0x7f950013f710 2024-03-25T18:25:55.106 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.105+0000 7f94f6ffd640 1 -- 172.21.15.97:0/3221149025 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2863224146 0 0) 0x7f95000ab920 con 0x7f95000a98f0 2024-03-25T18:25:55.106 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.105+0000 7f94f6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:25:55.106 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.105+0000 7f94f6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:55.107 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.105+0000 7f94f6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:55.107 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.105+0000 7f94f6ffd640 10 monclient(hunting): my global_id is 4659 2024-03-25T18:25:55.107 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.105+0000 7f94f6ffd640 1 -- 172.21.15.97:0/3221149025 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9500134e10 con 0x7f95000a98f0 2024-03-25T18:25:55.111 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.109+0000 7f94f6ffd640 1 -- 172.21.15.97:0/3221149025 <== mon.2 v1:172.21.15.97:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3231776493 0 0) 0x7f9500135140 con 0x7f950013f710 2024-03-25T18:25:55.111 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.109+0000 7f94f6ffd640 1 -- 172.21.15.97:0/3221149025 --> v1:172.21.15.97:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f95000ab920 con 0x7f950013f710 2024-03-25T18:25:55.111 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.109+0000 7f94f6ffd640 1 -- 172.21.15.97:0/3221149025 <== mon.2 v1:172.21.15.97:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f94f0001930 con 0x7f950013f710 2024-03-25T18:25:55.111 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.109+0000 7f94f6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:55.111 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.109+0000 7f94f6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:25:55.111 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.109+0000 7f94f6ffd640 10 monclient(hunting): dump: 2024-03-25T18:25:55.111 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:25:55.111 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:25:55.112 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:25:55.112 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:25:55.112 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:55.112 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:25:55.112 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:25:55.112 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:25:55.112 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:25:55.112 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:25:55.112 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.109+0000 7f94f6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:25:55.112 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.109+0000 7f94f6ffd640 1 -- 172.21.15.97:0/3221149025 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 930836224 0 0) 0x7f94e4004eb0 con 0x7f95001430c0 2024-03-25T18:25:55.112 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.109+0000 7f94f6ffd640 1 -- 172.21.15.97:0/3221149025 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9500135140 con 0x7f95001430c0 2024-03-25T18:25:55.112 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.109+0000 7f94f6ffd640 1 -- 172.21.15.97:0/3221149025 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f94f8001ec0 con 0x7f95001430c0 2024-03-25T18:25:55.112 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.109+0000 7f94f6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:55.112 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.109+0000 7f94f6ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:25:55.113 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.109+0000 7f94f6ffd640 10 monclient(hunting): dump: 2024-03-25T18:25:55.113 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:25:55.113 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:25:55.113 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:25:55.113 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:25:55.113 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:55.113 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:25:55.113 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:25:55.113 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:25:55.113 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:25:55.113 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:25:55.114 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.111+0000 7f94f6ffd640 1 -- 172.21.15.97:0/3221149025 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1952464597 0 0) 0x7f9500134e10 con 0x7f95000a98f0 2024-03-25T18:25:55.114 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.112+0000 7f94f6ffd640 1 -- 172.21.15.97:0/3221149025 --> v1:172.21.15.97:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f94e4004eb0 con 0x7f95000a98f0 2024-03-25T18:25:55.114 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.112+0000 7f94f6ffd640 1 -- 172.21.15.97:0/3221149025 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f94fc001830 con 0x7f95000a98f0 2024-03-25T18:25:55.114 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.112+0000 7f94f6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:55.114 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.112+0000 7f94f6ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:25:55.114 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.112+0000 7f94f6ffd640 10 monclient(hunting): dump: 2024-03-25T18:25:55.114 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:25:55.115 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:25:55.115 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:25:55.115 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:25:55.115 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:55.115 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:25:55.115 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:25:55.115 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:25:55.115 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:25:55.115 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:25:55.118 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.116+0000 7f94f6ffd640 1 -- 172.21.15.97:0/3221149025 <== mon.2 v1:172.21.15.97:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3385459017 0 0) 0x7f95000ab920 con 0x7f950013f710 2024-03-25T18:25:55.118 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.116+0000 7f94f6ffd640 1 -- 172.21.15.97:0/3221149025 >> v1:172.21.15.98:6789/0 conn(0x7f95001430c0 legacy=0x7f9500135660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:55.118 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.116+0000 7f94f6ffd640 1 -- 172.21.15.97:0/3221149025 >> v1:172.21.15.97:6789/0 conn(0x7f95000a98f0 legacy=0x7f95001385e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:55.118 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.116+0000 7f94f6ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:25:55.119 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.116+0000 7f94f6ffd640 1 monclient: found mon.c 2024-03-25T18:25:55.119 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.116+0000 7f94f6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:55.119 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.116+0000 7f94f6ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.97:6790/0 2024-03-25T18:25:55.119 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.116+0000 7f94f6ffd640 1 -- 172.21.15.97:0/3221149025 --> v1:172.21.15.97:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9500137840 con 0x7f950013f710 2024-03-25T18:25:55.119 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.116+0000 7f94f6ffd640 10 monclient: _finish_auth 0 2024-03-25T18:25:55.119 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.116+0000 7f94f6ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:25:55.119 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.116+0000 7f94f6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:55.119 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.116+0000 7f95082a8640 5 monclient: authenticate success, global_id 4643 2024-03-25T18:25:55.119 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.116+0000 7f95082a8640 10 monclient: _renew_subs 2024-03-25T18:25:55.119 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.116+0000 7f95082a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.97:6790/0 2024-03-25T18:25:55.119 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.116+0000 7f95082a8640 1 -- 172.21.15.97:0/3221149025 --> v1:172.21.15.97:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9500137a90 con 0x7f950013f710 2024-03-25T18:25:55.119 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.116+0000 7f95082a8640 10 client.4643.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:25:55.119 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.116+0000 7f95082a8640 10 monclient: _renew_subs 2024-03-25T18:25:55.120 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.116+0000 7f95082a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.97:6790/0 2024-03-25T18:25:55.120 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.116+0000 7f95082a8640 1 -- 172.21.15.97:0/3221149025 --> v1:172.21.15.97:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f95001347e0 con 0x7f950013f710 2024-03-25T18:25:55.120 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.116+0000 7f95082a8640 1 librados: init done 2024-03-25T18:25:55.120 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.117+0000 7f95082a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:55.120 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.117+0000 7f95082a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:55.120 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.117+0000 7f95082a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.97:6790/0 2024-03-25T18:25:55.121 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.117+0000 7f95082a8640 1 -- 172.21.15.97:0/3221149025 --> v1:172.21.15.97:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9500134fa0 con 0x7f950013f710 2024-03-25T18:25:55.418 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.416+0000 7f94f6ffd640 1 -- 172.21.15.97:0/3221149025 <== mon.2 v1:172.21.15.97:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f94f0001d60 con 0x7f950013f710 2024-03-25T18:25:55.418 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.416+0000 7f94f6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:55.419 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.416+0000 7f94f6ffd640 1 -- 172.21.15.97:0/3221149025 <== mon.2 v1:172.21.15.97:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f94f00035f0 con 0x7f950013f710 2024-03-25T18:25:55.419 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.416+0000 7f94f6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:55.419 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.416+0000 7f94f6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:25:55.419 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.416+0000 7f94f6ffd640 10 monclient: dump: 2024-03-25T18:25:55.419 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:25:55.419 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:25:55.419 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:25:55.419 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:25:55.419 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:55.419 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:25:55.419 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:25:55.419 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:25:55.419 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:25:55.420 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:25:55.420 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.416+0000 7f9504841640 4 set_mon_vals no callback set 2024-03-25T18:25:55.420 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.417+0000 7f94f6ffd640 1 -- 172.21.15.97:0/3221149025 <== mon.2 v1:172.21.15.97:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2377622137 0 0) 0x7f94f0019640 con 0x7f950013f710 2024-03-25T18:25:55.420 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.418+0000 7f9506044640 10 monclient: get_auth_request con 0x7f94e406e0f0 auth_method 0 2024-03-25T18:25:55.420 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.418+0000 7f94f6ffd640 10 client.4643.objecter ms_handle_connect 0x7f94e406e0f0 2024-03-25T18:25:55.424 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.422+0000 7f94f6ffd640 1 -- 172.21.15.97:0/3221149025 <== mon.2 v1:172.21.15.97:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 430711918 0 0) 0x7f94f00532e0 con 0x7f950013f710 2024-03-25T18:25:55.424 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.422+0000 7f94f6ffd640 10 client.4643.objecter ms_dispatch 0x7f9500000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:25:55.424 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.422+0000 7f94f6ffd640 3 client.4643.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:25:55.425 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.422+0000 7f94f6ffd640 3 client.4643.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:25:55.425 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.423+0000 7f94f6ffd640 20 client.4643.objecter dump_active .. 0 homeless 2024-03-25T18:25:55.425 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.423+0000 7f94f6ffd640 1 -- 172.21.15.97:0/3221149025 <== mon.2 v1:172.21.15.97:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f94f004ec20 con 0x7f950013f710 2024-03-25T18:25:55.425 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.423+0000 7f94f6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:55.425 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.423+0000 7f94f6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:25:55.644 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.642+0000 7f95082a8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:55.644 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.642+0000 7f95082a8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:55.644 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.642+0000 7f95082a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.97:6790/0 2024-03-25T18:25:55.644 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.642+0000 7f95082a8640 1 -- 172.21.15.97:0/3221149025 --> v1:172.21.15.97:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f950014f320 con 0x7f950013f710 2024-03-25T18:25:55.677 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.675+0000 7f94f6ffd640 1 -- 172.21.15.97:0/3221149025 <== mon.2 v1:172.21.15.97:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2169427412) 0x7f94f0050d20 con 0x7f950013f710 2024-03-25T18:25:55.677 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.675+0000 7f94f6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:55.677 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.675+0000 7f94f6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:25:55.677 INFO:teuthology.misc.health.smithi097.stdout: 2024-03-25T18:25:55.677 INFO:teuthology.misc.health.smithi097.stdout:{"epoch":10,"fsid":"4c3d472c-79b8-486b-9e3d-2738d4306754","created":"2024-03-25T18:16:14.737309+0000","modified":"2024-03-25T18:16:20.357492+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:20.357492+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cb5152b9-262f-45c3-94b8-18d0e81df558","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6daf8793-0838-473d-a7d6-9a0d2894c75d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"136f1493-3b9c-4f43-a2e7-641482d69dae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"397d538c-4f64-4daf-9e8b-662d09f91327","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4f1df08-663f-4bda-93f8-3878ab7c24b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c4b18d70-a3ce-4b15-9b70-eae5525bd761","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8dd4033a-de0e-4f8a-96dc-59f3c1f825b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0077722e-ffa6-4a3d-9891-929d9869b286","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:25:55.680 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.677+0000 7f95082a8640 10 librados: watch_flush enter 2024-03-25T18:25:55.680 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.677+0000 7f95082a8640 10 librados: watch_flush exit 2024-03-25T18:25:55.680 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.677+0000 7f95082a8640 20 client.4643.objecter shutdown clearing up homeless session... 2024-03-25T18:25:55.680 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.677+0000 7f95082a8640 10 client.4643.objecter successfully canceled tick 2024-03-25T18:25:55.680 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.677+0000 7f95082a8640 1 -- 172.21.15.97:0/3221149025 >> v1:172.21.15.98:6800/800387573 conn(0x7f94e406e0f0 legacy=0x7f94e40705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:55.680 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.677+0000 7f95082a8640 10 monclient: shutdown 2024-03-25T18:25:55.680 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.677+0000 7f95082a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:55.680 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.677+0000 7f95082a8640 1 -- 172.21.15.97:0/3221149025 >> v1:172.21.15.97:6790/0 conn(0x7f950013f710 legacy=0x7f9500133ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:55.680 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.677+0000 7f95082a8640 1 -- 172.21.15.97:0/3221149025 shutdown_connections 2024-03-25T18:25:55.680 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.678+0000 7f95082a8640 1 -- 172.21.15.97:0/3221149025 >> 172.21.15.97:0/3221149025 conn(0x7f95000b7ea0 msgr2=0x7f950013ca40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:55.680 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.678+0000 7f95082a8640 1 -- 172.21.15.97:0/3221149025 shutdown_connections 2024-03-25T18:25:55.680 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.678+0000 7f95082a8640 1 -- 172.21.15.97:0/3221149025 wait complete. 2024-03-25T18:25:55.680 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:25:55.678+0000 7f95082a8640 1 librados: shutdown 2024-03-25T18:25:55.690 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:25:58.802 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:10:00.840 34931== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:58.809 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:10:00.847 34935== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:58.812 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:10:00.850 34931== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:58.861 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:10:00.899 34931== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:58.869 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:10:00.912 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:58.870 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:10:00.908 34935== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:58.876 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:10:00.919 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:58.889 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:10:00.928 34935== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:58.899 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:10:00.942 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:02.498 DEBUG:teuthology.orchestra.run.smithi097:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:26:02.615 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.614+0000 7f6878bc1640 10 monclient: get_monmap_and_config 2024-03-25T18:26:02.616 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.614+0000 7f6878bc1640 10 monclient: build_initial_monmap 2024-03-25T18:26:02.616 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.614+0000 7f6878bc1640 10 monclient: monmap: 2024-03-25T18:26:02.616 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:26:02.616 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:26:02.616 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:26:02.615493+0000 2024-03-25T18:26:02.616 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:26:02.615493+0000 2024-03-25T18:26:02.616 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:02.616 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:26:02.616 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:26:02.616 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T18:26:02.616 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-b 2024-03-25T18:26:02.616 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:26:02.618 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.614+0000 7f6878bc1640 1 Processor -- start 2024-03-25T18:26:02.618 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.614+0000 7f6878bc1640 1 -- start start 2024-03-25T18:26:02.618 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.614+0000 7f6878bc1640 10 monclient: init 2024-03-25T18:26:02.618 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.615+0000 7f6878bc1640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:02.618 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.615+0000 7f6878bc1640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:26:02.618 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.615+0000 7f6878bc1640 10 monclient(hunting): picked mon.noname-a con 0x7f68740a98f0 addr v1:172.21.15.97:6789/0 2024-03-25T18:26:02.618 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.615+0000 7f6878bc1640 10 monclient(hunting): picked mon.noname-b con 0x7f687413f710 addr v1:172.21.15.97:6790/0 2024-03-25T18:26:02.618 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.615+0000 7f6878bc1640 10 monclient(hunting): picked mon.noname-c con 0x7f68741430c0 addr v1:172.21.15.98:6789/0 2024-03-25T18:26:02.618 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.615+0000 7f6878bc1640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6874142520 con 0x7f68740a98f0 2024-03-25T18:26:02.618 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.615+0000 7f6878bc1640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68740b28a0 con 0x7f68741430c0 2024-03-25T18:26:02.618 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.615+0000 7f6878bc1640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68740b2a30 con 0x7f687413f710 2024-03-25T18:26:02.618 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.615+0000 7f6878bc1640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:02.618 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.615+0000 7f6878bc1640 10 monclient(hunting): authenticate will time out at 8174.891575s 2024-03-25T18:26:02.629 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.627+0000 7f6871d9b640 1 --1- >> v1:172.21.15.97:6790/0 conn(0x7f687413f710 0x7f68741429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6790/0 says I am v1:172.21.15.97:35734/0 (socket says 172.21.15.97:35734) 2024-03-25T18:26:02.629 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.627+0000 7f6871d9b640 1 -- 172.21.15.97:0/4172429956 learned_addr learned my addr 172.21.15.97:0/4172429956 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:26:02.641 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.639+0000 7f687159a640 1 -- 172.21.15.97:0/4172429956 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 795784711 0 0) 0x7f6874142520 con 0x7f68740a98f0 2024-03-25T18:26:02.641 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.639+0000 7f687159a640 10 monclient(hunting): _negotiate 2024-03-25T18:26:02.641 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.639+0000 7f687159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:02.641 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.639+0000 7f687159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:02.641 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.639+0000 7f687159a640 10 monclient(hunting): my global_id is 4662 2024-03-25T18:26:02.641 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.639+0000 7f687159a640 1 -- 172.21.15.97:0/4172429956 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f685c003520 con 0x7f68740a98f0 2024-03-25T18:26:02.650 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.648+0000 7f687159a640 1 -- 172.21.15.97:0/4172429956 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4246449672 0 0) 0x7f68740b28a0 con 0x7f68741430c0 2024-03-25T18:26:02.650 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.648+0000 7f687159a640 10 monclient(hunting): _negotiate 2024-03-25T18:26:02.650 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.648+0000 7f687159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:02.650 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.648+0000 7f687159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:02.650 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.648+0000 7f687159a640 10 monclient(hunting): my global_id is 4666 2024-03-25T18:26:02.650 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.648+0000 7f687159a640 1 -- 172.21.15.97:0/4172429956 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6874142520 con 0x7f68741430c0 2024-03-25T18:26:02.654 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.652+0000 7f687159a640 1 -- 172.21.15.97:0/4172429956 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1164540653 0 0) 0x7f68740b2a30 con 0x7f687413f710 2024-03-25T18:26:02.654 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.652+0000 7f687159a640 10 monclient(hunting): _negotiate 2024-03-25T18:26:02.654 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.652+0000 7f687159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:02.654 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.652+0000 7f687159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:02.654 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.652+0000 7f687159a640 10 monclient(hunting): my global_id is 4646 2024-03-25T18:26:02.654 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.652+0000 7f687159a640 1 -- 172.21.15.97:0/4172429956 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68740b28a0 con 0x7f687413f710 2024-03-25T18:26:02.656 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.654+0000 7f687159a640 1 -- 172.21.15.97:0/4172429956 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3068151318 0 0) 0x7f685c003520 con 0x7f68740a98f0 2024-03-25T18:26:02.656 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.654+0000 7f687159a640 1 -- 172.21.15.97:0/4172429956 >> v1:172.21.15.97:6790/0 conn(0x7f687413f710 legacy=0x7f68741429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:02.656 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.654+0000 7f687159a640 1 -- 172.21.15.97:0/4172429956 >> v1:172.21.15.98:6789/0 conn(0x7f68741430c0 legacy=0x7f687414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:02.656 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.654+0000 7f687159a640 10 monclient: _finish_hunting 0 2024-03-25T18:26:02.656 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.654+0000 7f687159a640 1 monclient: found mon.noname-a 2024-03-25T18:26:02.657 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.654+0000 7f687159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:02.657 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.654+0000 7f687159a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.97:6789/0 2024-03-25T18:26:02.657 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.654+0000 7f687159a640 1 -- 172.21.15.97:0/4172429956 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68740b44c0 con 0x7f68740a98f0 2024-03-25T18:26:02.657 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.654+0000 7f687159a640 10 monclient: _finish_auth 0 2024-03-25T18:26:02.657 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.654+0000 7f687159a640 10 monclient: _check_auth_tickets 2024-03-25T18:26:02.657 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.654+0000 7f687159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:02.657 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.654+0000 7f687159a640 1 -- 172.21.15.97:0/4172429956 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f6864001690 con 0x7f68740a98f0 2024-03-25T18:26:02.657 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.654+0000 7f687159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:02.657 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.654+0000 7f687159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:26:02.657 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.654+0000 7f687159a640 10 monclient: dump: 2024-03-25T18:26:02.657 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:26:02.657 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:26:02.657 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:26:02.657 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:26:02.658 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:02.658 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:26:02.658 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:26:02.658 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:26:02.658 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:26:02.658 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:26:02.658 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.654+0000 7f6878bc1640 5 monclient: authenticate success, global_id 4662 2024-03-25T18:26:02.658 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.654+0000 7f6878bc1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:26:02.672 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.670+0000 7f687159a640 1 -- 172.21.15.97:0/4172429956 <== mon.0 v1:172.21.15.97:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f68640017f0 con 0x7f68740a98f0 2024-03-25T18:26:02.672 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.670+0000 7f687159a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:02.672 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.670+0000 7f687159a640 1 -- 172.21.15.97:0/4172429956 <== mon.0 v1:172.21.15.97:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f6864001c00 con 0x7f68740a98f0 2024-03-25T18:26:02.672 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.670+0000 7f687159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:02.672 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.670+0000 7f687159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:26:02.672 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.670+0000 7f687159a640 10 monclient: dump: 2024-03-25T18:26:02.672 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:26:02.672 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:26:02.672 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:26:02.672 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:26:02.673 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:02.673 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:26:02.673 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:26:02.673 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:26:02.673 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:26:02.673 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:26:02.673 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.670+0000 7f6878bc1640 10 monclient: get_monmap_and_config success 2024-03-25T18:26:02.673 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.670+0000 7f6878bc1640 4 set_mon_vals no callback set 2024-03-25T18:26:02.675 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.671+0000 7f6878bc1640 10 monclient: shutdown 2024-03-25T18:26:02.675 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.671+0000 7f6878bc1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:02.675 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.671+0000 7f6878bc1640 1 -- 172.21.15.97:0/4172429956 >> v1:172.21.15.97:6789/0 conn(0x7f68740a98f0 legacy=0x7f68740a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:02.675 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.671+0000 7f6878bc1640 1 -- 172.21.15.97:0/4172429956 shutdown_connections 2024-03-25T18:26:02.675 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.671+0000 7f6878bc1640 1 -- 172.21.15.97:0/4172429956 >> 172.21.15.97:0/4172429956 conn(0x7f68740b7ea0 msgr2=0x7f68740b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:02.675 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.671+0000 7f6878bc1640 1 -- 172.21.15.97:0/4172429956 shutdown_connections 2024-03-25T18:26:02.675 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.671+0000 7f6878bc1640 1 -- 172.21.15.97:0/4172429956 wait complete. 2024-03-25T18:26:02.675 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.672+0000 7f6878bc1640 10 monclient: build_initial_monmap 2024-03-25T18:26:02.675 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.672+0000 7f6878bc1640 10 monclient: monmap: 2024-03-25T18:26:02.675 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:26:02.675 INFO:teuthology.misc.health.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:26:02.675 INFO:teuthology.misc.health.smithi097.stderr:last_changed 0.000000 2024-03-25T18:26:02.676 INFO:teuthology.misc.health.smithi097.stderr:created 0.000000 2024-03-25T18:26:02.676 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:02.676 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 1 2024-03-25T18:26:02.676 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:26:02.676 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T18:26:02.676 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-c 2024-03-25T18:26:02.676 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:26:02.676 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.673+0000 7f6878bc1640 1 librados: starting msgr at 2024-03-25T18:26:02.676 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.673+0000 7f6878bc1640 1 librados: starting objecter 2024-03-25T18:26:02.676 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.673+0000 7f6878bc1640 1 Processor -- start 2024-03-25T18:26:02.676 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.673+0000 7f6878bc1640 1 -- start start 2024-03-25T18:26:02.676 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.673+0000 7f6878bc1640 1 librados: setting wanted keys 2024-03-25T18:26:02.677 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.673+0000 7f6878bc1640 1 librados: calling monclient init 2024-03-25T18:26:02.677 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.673+0000 7f6878bc1640 10 monclient: init 2024-03-25T18:26:02.678 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.674+0000 7f6878bc1640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:02.678 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.674+0000 7f6878bc1640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:26:02.678 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.674+0000 7f6878bc1640 10 monclient(hunting): picked mon.noname-c con 0x7f68740a98f0 addr v1:172.21.15.97:6790/0 2024-03-25T18:26:02.678 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.674+0000 7f6878bc1640 10 monclient(hunting): picked mon.noname-a con 0x7f687413f710 addr v1:172.21.15.97:6789/0 2024-03-25T18:26:02.678 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.674+0000 7f6878bc1640 10 monclient(hunting): picked mon.noname-b con 0x7f68741430c0 addr v1:172.21.15.98:6789/0 2024-03-25T18:26:02.678 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.674+0000 7f6878bc1640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68741390e0 con 0x7f687413f710 2024-03-25T18:26:02.678 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.674+0000 7f6878bc1640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68741db0a0 con 0x7f68741430c0 2024-03-25T18:26:02.678 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.674+0000 7f6878bc1640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6874137580 con 0x7f68740a98f0 2024-03-25T18:26:02.678 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.674+0000 7f6878bc1640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:02.678 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.674+0000 7f6878bc1640 10 monclient(hunting): authenticate will time out at 8174.950320s 2024-03-25T18:26:02.680 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.679+0000 7f6871d9b640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f687413f710 0x7f68741369a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:46590/0 (socket says 172.21.15.97:46590) 2024-03-25T18:26:02.680 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.679+0000 7f6871d9b640 1 -- 172.21.15.97:0/2613846284 learned_addr learned my addr 172.21.15.97:0/2613846284 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:26:02.683 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.682+0000 7f6862ffd640 10 client.?.objecter ms_handle_connect 0x7f68741430c0 2024-03-25T18:26:02.683 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.682+0000 7f6862ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:02.684 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.682+0000 7f6862ffd640 10 client.?.objecter ms_handle_connect 0x7f687413f710 2024-03-25T18:26:02.684 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.682+0000 7f6862ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:02.691 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.689+0000 7f6862ffd640 10 client.?.objecter ms_handle_connect 0x7f68740a98f0 2024-03-25T18:26:02.691 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.689+0000 7f6862ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:02.698 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.696+0000 7f6862ffd640 1 -- 172.21.15.97:0/2613846284 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 836236301 0 0) 0x7f68741390e0 con 0x7f687413f710 2024-03-25T18:26:02.698 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.696+0000 7f6862ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:26:02.698 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.696+0000 7f6862ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:02.698 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.696+0000 7f6862ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:02.698 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.696+0000 7f6862ffd640 10 monclient(hunting): my global_id is 4665 2024-03-25T18:26:02.698 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.696+0000 7f6862ffd640 1 -- 172.21.15.97:0/2613846284 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6854004eb0 con 0x7f687413f710 2024-03-25T18:26:02.704 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.702+0000 7f6862ffd640 1 -- 172.21.15.97:0/2613846284 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1517385392 0 0) 0x7f6874137580 con 0x7f68740a98f0 2024-03-25T18:26:02.704 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.702+0000 7f6862ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:26:02.704 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.702+0000 7f6862ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:02.704 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.702+0000 7f6862ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:02.704 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.702+0000 7f6862ffd640 10 monclient(hunting): my global_id is 4649 2024-03-25T18:26:02.704 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.702+0000 7f6862ffd640 1 -- 172.21.15.97:0/2613846284 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68741390e0 con 0x7f68740a98f0 2024-03-25T18:26:02.705 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.703+0000 7f6862ffd640 1 -- 172.21.15.97:0/2613846284 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 799568725 0 0) 0x7f68741db0a0 con 0x7f68741430c0 2024-03-25T18:26:02.705 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.703+0000 7f6862ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:26:02.705 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.703+0000 7f6862ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:02.705 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.703+0000 7f6862ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:02.705 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.703+0000 7f6862ffd640 10 monclient(hunting): my global_id is 4669 2024-03-25T18:26:02.705 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.703+0000 7f6862ffd640 1 -- 172.21.15.97:0/2613846284 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6874137580 con 0x7f68741430c0 2024-03-25T18:26:02.726 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.724+0000 7f6862ffd640 1 -- 172.21.15.97:0/2613846284 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 536499222 0 0) 0x7f6874137580 con 0x7f68741430c0 2024-03-25T18:26:02.726 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.724+0000 7f6862ffd640 1 -- 172.21.15.97:0/2613846284 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f68741db0a0 con 0x7f68741430c0 2024-03-25T18:26:02.726 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.724+0000 7f6862ffd640 1 -- 172.21.15.97:0/2613846284 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f686c001bd0 con 0x7f68741430c0 2024-03-25T18:26:02.726 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.724+0000 7f6862ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:02.726 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.724+0000 7f6862ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:26:02.726 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.724+0000 7f6862ffd640 10 monclient(hunting): dump: 2024-03-25T18:26:02.727 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:26:02.727 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:26:02.727 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:26:02.727 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:26:02.727 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:02.727 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:26:02.727 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:26:02.727 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:26:02.727 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:26:02.727 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:26:02.727 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.724+0000 7f6862ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:26:02.728 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.725+0000 7f6862ffd640 1 -- 172.21.15.97:0/2613846284 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2578648305 0 0) 0x7f6854004eb0 con 0x7f687413f710 2024-03-25T18:26:02.728 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.726+0000 7f6862ffd640 1 -- 172.21.15.97:0/2613846284 --> v1:172.21.15.97:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6874137580 con 0x7f687413f710 2024-03-25T18:26:02.728 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.726+0000 7f6862ffd640 1 -- 172.21.15.97:0/2613846284 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f6868001910 con 0x7f687413f710 2024-03-25T18:26:02.728 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.726+0000 7f6862ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:02.728 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.726+0000 7f6862ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:26:02.728 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.726+0000 7f6862ffd640 10 monclient(hunting): dump: 2024-03-25T18:26:02.728 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:26:02.728 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:26:02.729 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:26:02.729 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:26:02.729 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:02.729 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:26:02.729 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:26:02.729 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:26:02.729 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:26:02.729 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:26:02.733 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.731+0000 7f6862ffd640 1 -- 172.21.15.97:0/2613846284 <== mon.2 v1:172.21.15.97:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2036274109 0 0) 0x7f68741390e0 con 0x7f68740a98f0 2024-03-25T18:26:02.733 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.731+0000 7f6862ffd640 1 -- 172.21.15.97:0/2613846284 --> v1:172.21.15.97:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6854004eb0 con 0x7f68740a98f0 2024-03-25T18:26:02.733 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.731+0000 7f6862ffd640 1 -- 172.21.15.97:0/2613846284 <== mon.2 v1:172.21.15.97:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f6864001b00 con 0x7f68740a98f0 2024-03-25T18:26:02.733 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.731+0000 7f6862ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:02.733 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.731+0000 7f6862ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:26:02.733 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.731+0000 7f6862ffd640 10 monclient(hunting): dump: 2024-03-25T18:26:02.733 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:26:02.734 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:26:02.734 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:26:02.734 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:26:02.734 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:02.734 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:26:02.734 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:26:02.734 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:26:02.734 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:26:02.734 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:26:02.743 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.741+0000 7f6862ffd640 1 -- 172.21.15.97:0/2613846284 <== mon.2 v1:172.21.15.97:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 236213658 0 0) 0x7f6854004eb0 con 0x7f68740a98f0 2024-03-25T18:26:02.743 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.741+0000 7f6862ffd640 1 -- 172.21.15.97:0/2613846284 >> v1:172.21.15.98:6789/0 conn(0x7f68741430c0 legacy=0x7f6874133090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:02.744 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.741+0000 7f6862ffd640 1 -- 172.21.15.97:0/2613846284 >> v1:172.21.15.97:6789/0 conn(0x7f687413f710 legacy=0x7f68741369a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:02.744 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.741+0000 7f6862ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:26:02.744 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.741+0000 7f6862ffd640 1 monclient: found mon.c 2024-03-25T18:26:02.744 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.741+0000 7f6862ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:02.744 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.741+0000 7f6862ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.97:6790/0 2024-03-25T18:26:02.744 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.741+0000 7f6862ffd640 1 -- 172.21.15.97:0/2613846284 --> v1:172.21.15.97:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68741370b0 con 0x7f68740a98f0 2024-03-25T18:26:02.744 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.741+0000 7f6862ffd640 10 monclient: _finish_auth 0 2024-03-25T18:26:02.744 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.741+0000 7f6862ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:26:02.744 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.741+0000 7f6862ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:02.744 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.741+0000 7f6878bc1640 5 monclient: authenticate success, global_id 4649 2024-03-25T18:26:02.744 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.741+0000 7f6878bc1640 10 monclient: _renew_subs 2024-03-25T18:26:02.744 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.741+0000 7f6878bc1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.97:6790/0 2024-03-25T18:26:02.744 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.741+0000 7f6878bc1640 1 -- 172.21.15.97:0/2613846284 --> v1:172.21.15.97:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68741337a0 con 0x7f68740a98f0 2024-03-25T18:26:02.744 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.741+0000 7f6878bc1640 10 client.4649.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:26:02.744 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.741+0000 7f6878bc1640 10 monclient: _renew_subs 2024-03-25T18:26:02.744 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.741+0000 7f6878bc1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.97:6790/0 2024-03-25T18:26:02.745 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.741+0000 7f6878bc1640 1 -- 172.21.15.97:0/2613846284 --> v1:172.21.15.97:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f68740b7170 con 0x7f68740a98f0 2024-03-25T18:26:02.745 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.741+0000 7f6878bc1640 1 librados: init done 2024-03-25T18:26:02.745 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.742+0000 7f6878bc1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:02.745 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.742+0000 7f6878bc1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:02.745 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.742+0000 7f6878bc1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.97:6790/0 2024-03-25T18:26:02.745 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:02.742+0000 7f6878bc1640 1 -- 172.21.15.97:0/2613846284 --> v1:172.21.15.97:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68740627b0 con 0x7f68740a98f0 2024-03-25T18:26:03.040 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:03.038+0000 7f6862ffd640 1 -- 172.21.15.97:0/2613846284 <== mon.2 v1:172.21.15.97:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6864001c60 con 0x7f68740a98f0 2024-03-25T18:26:03.040 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:03.038+0000 7f6862ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:03.041 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:03.038+0000 7f6862ffd640 1 -- 172.21.15.97:0/2613846284 <== mon.2 v1:172.21.15.97:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f6864003a40 con 0x7f68740a98f0 2024-03-25T18:26:03.041 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:03.038+0000 7f6862ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:03.041 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:03.038+0000 7f6862ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:26:03.041 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:03.038+0000 7f6862ffd640 10 monclient: dump: 2024-03-25T18:26:03.041 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:26:03.041 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:26:03.041 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:26:03.041 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:26:03.041 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:03.041 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:26:03.041 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:26:03.041 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:26:03.041 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:26:03.042 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:26:03.042 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:03.038+0000 7f6870d99640 4 set_mon_vals no callback set 2024-03-25T18:26:03.042 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:03.039+0000 7f6862ffd640 1 -- 172.21.15.97:0/2613846284 <== mon.2 v1:172.21.15.97:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2377622137 0 0) 0x7f6864019a10 con 0x7f68740a98f0 2024-03-25T18:26:03.042 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:03.040+0000 7f6871d9b640 10 monclient: get_auth_request con 0x7f685406e0c0 auth_method 0 2024-03-25T18:26:03.042 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:03.040+0000 7f6862ffd640 10 client.4649.objecter ms_handle_connect 0x7f685406e0c0 2024-03-25T18:26:03.049 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:03.047+0000 7f6862ffd640 1 -- 172.21.15.97:0/2613846284 <== mon.2 v1:172.21.15.97:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 430711918 0 0) 0x7f6864053620 con 0x7f68740a98f0 2024-03-25T18:26:03.049 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:03.047+0000 7f6862ffd640 10 client.4649.objecter ms_dispatch 0x7f6874000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:26:03.049 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:03.047+0000 7f6862ffd640 3 client.4649.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:26:03.049 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:03.047+0000 7f6862ffd640 3 client.4649.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:26:03.049 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:03.047+0000 7f6862ffd640 20 client.4649.objecter dump_active .. 0 homeless 2024-03-25T18:26:03.049 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:03.047+0000 7f6862ffd640 1 -- 172.21.15.97:0/2613846284 <== mon.2 v1:172.21.15.97:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f686404ef60 con 0x7f68740a98f0 2024-03-25T18:26:03.049 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:03.047+0000 7f6862ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:03.049 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:03.047+0000 7f6862ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:26:03.283 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:03.281+0000 7f6878bc1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:03.283 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:03.281+0000 7f6878bc1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:03.283 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:03.281+0000 7f6878bc1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.97:6790/0 2024-03-25T18:26:03.283 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:03.281+0000 7f6878bc1640 1 -- 172.21.15.97:0/2613846284 --> v1:172.21.15.97:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f68740b7470 con 0x7f68740a98f0 2024-03-25T18:26:03.326 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:03.324+0000 7f6862ffd640 1 -- 172.21.15.97:0/2613846284 <== mon.2 v1:172.21.15.97:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2169427412) 0x7f6864051060 con 0x7f68740a98f0 2024-03-25T18:26:03.326 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:03.324+0000 7f6862ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:03.326 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:03.324+0000 7f6862ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:26:03.327 INFO:teuthology.misc.health.smithi097.stdout: 2024-03-25T18:26:03.327 INFO:teuthology.misc.health.smithi097.stdout:{"epoch":10,"fsid":"4c3d472c-79b8-486b-9e3d-2738d4306754","created":"2024-03-25T18:16:14.737309+0000","modified":"2024-03-25T18:16:20.357492+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:20.357492+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cb5152b9-262f-45c3-94b8-18d0e81df558","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6daf8793-0838-473d-a7d6-9a0d2894c75d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"136f1493-3b9c-4f43-a2e7-641482d69dae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"397d538c-4f64-4daf-9e8b-662d09f91327","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4f1df08-663f-4bda-93f8-3878ab7c24b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c4b18d70-a3ce-4b15-9b70-eae5525bd761","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8dd4033a-de0e-4f8a-96dc-59f3c1f825b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0077722e-ffa6-4a3d-9891-929d9869b286","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:26:03.328 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:03.326+0000 7f6878bc1640 10 librados: watch_flush enter 2024-03-25T18:26:03.329 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:03.326+0000 7f6878bc1640 10 librados: watch_flush exit 2024-03-25T18:26:03.329 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:03.326+0000 7f6878bc1640 20 client.4649.objecter shutdown clearing up homeless session... 2024-03-25T18:26:03.329 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:03.326+0000 7f6878bc1640 10 client.4649.objecter successfully canceled tick 2024-03-25T18:26:03.329 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:03.326+0000 7f6878bc1640 1 -- 172.21.15.97:0/2613846284 >> v1:172.21.15.98:6800/800387573 conn(0x7f685406e0c0 legacy=0x7f6854070580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:03.329 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:03.326+0000 7f6878bc1640 10 monclient: shutdown 2024-03-25T18:26:03.329 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:03.326+0000 7f6878bc1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:03.329 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:03.326+0000 7f6878bc1640 1 -- 172.21.15.97:0/2613846284 >> v1:172.21.15.97:6790/0 conn(0x7f68740a98f0 legacy=0x7f68741da990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:03.329 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:03.326+0000 7f6878bc1640 1 -- 172.21.15.97:0/2613846284 shutdown_connections 2024-03-25T18:26:03.329 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:03.326+0000 7f6878bc1640 1 -- 172.21.15.97:0/2613846284 >> 172.21.15.97:0/2613846284 conn(0x7f68740b7ea0 msgr2=0x7f687413ca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:03.329 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:03.327+0000 7f6878bc1640 1 -- 172.21.15.97:0/2613846284 shutdown_connections 2024-03-25T18:26:03.329 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:03.327+0000 7f6878bc1640 1 -- 172.21.15.97:0/2613846284 wait complete. 2024-03-25T18:26:03.329 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:03.327+0000 7f6878bc1640 1 librados: shutdown 2024-03-25T18:26:03.341 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:26:10.147 DEBUG:teuthology.orchestra.run.smithi097:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:26:10.256 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.255+0000 7f4efa3c7640 10 monclient: get_monmap_and_config 2024-03-25T18:26:10.256 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.255+0000 7f4efa3c7640 10 monclient: build_initial_monmap 2024-03-25T18:26:10.256 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.255+0000 7f4efa3c7640 10 monclient: monmap: 2024-03-25T18:26:10.256 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:26:10.256 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:26:10.256 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:26:10.255815+0000 2024-03-25T18:26:10.257 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:26:10.255815+0000 2024-03-25T18:26:10.257 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:10.257 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:26:10.257 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:26:10.257 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T18:26:10.257 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-b 2024-03-25T18:26:10.257 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:26:10.258 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.256+0000 7f4efa3c7640 1 Processor -- start 2024-03-25T18:26:10.259 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.256+0000 7f4efa3c7640 1 -- start start 2024-03-25T18:26:10.259 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.256+0000 7f4efa3c7640 10 monclient: init 2024-03-25T18:26:10.259 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.257+0000 7f4efa3c7640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:10.259 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.257+0000 7f4efa3c7640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:26:10.259 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.257+0000 7f4efa3c7640 10 monclient(hunting): picked mon.noname-b con 0x7f4ef40b4d60 addr v1:172.21.15.97:6790/0 2024-03-25T18:26:10.259 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.257+0000 7f4efa3c7640 10 monclient(hunting): picked mon.noname-a con 0x7f4ef40b1730 addr v1:172.21.15.97:6789/0 2024-03-25T18:26:10.259 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.257+0000 7f4efa3c7640 10 monclient(hunting): picked mon.noname-c con 0x7f4ef414be90 addr v1:172.21.15.98:6789/0 2024-03-25T18:26:10.259 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.257+0000 7f4efa3c7640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ef40af940 con 0x7f4ef40b1730 2024-03-25T18:26:10.259 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.257+0000 7f4efa3c7640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ef40b0a70 con 0x7f4ef414be90 2024-03-25T18:26:10.259 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.257+0000 7f4efa3c7640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ef40b0d50 con 0x7f4ef40b4d60 2024-03-25T18:26:10.259 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.257+0000 7f4efa3c7640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:10.259 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.257+0000 7f4efa3c7640 10 monclient(hunting): authenticate will time out at 8182.532323s 2024-03-25T18:26:10.266 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.265+0000 7f4ef3fff640 1 --1- >> v1:172.21.15.97:6790/0 conn(0x7f4ef40b4d60 0x7f4ef413f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6790/0 says I am v1:172.21.15.97:40532/0 (socket says 172.21.15.97:40532) 2024-03-25T18:26:10.266 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.265+0000 7f4ef3fff640 1 -- 172.21.15.97:0/602066020 learned_addr learned my addr 172.21.15.97:0/602066020 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:26:10.275 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.274+0000 7f4ef37fe640 1 -- 172.21.15.97:0/602066020 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1885793183 0 0) 0x7f4ef40b0d50 con 0x7f4ef40b4d60 2024-03-25T18:26:10.275 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.274+0000 7f4ef37fe640 10 monclient(hunting): _negotiate 2024-03-25T18:26:10.275 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.274+0000 7f4ef37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:10.275 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.274+0000 7f4ef37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:10.275 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.274+0000 7f4ef37fe640 10 monclient(hunting): my global_id is 4652 2024-03-25T18:26:10.275 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.274+0000 7f4ef37fe640 1 -- 172.21.15.97:0/602066020 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ee0003520 con 0x7f4ef40b4d60 2024-03-25T18:26:10.285 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.284+0000 7f4ef37fe640 1 -- 172.21.15.97:0/602066020 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 584442646 0 0) 0x7f4ef40b0a70 con 0x7f4ef414be90 2024-03-25T18:26:10.285 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.284+0000 7f4ef37fe640 10 monclient(hunting): _negotiate 2024-03-25T18:26:10.285 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.284+0000 7f4ef37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:10.285 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.284+0000 7f4ef37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:10.285 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.284+0000 7f4ef37fe640 10 monclient(hunting): my global_id is 4672 2024-03-25T18:26:10.285 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.284+0000 7f4ef37fe640 1 -- 172.21.15.97:0/602066020 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ef40b0d50 con 0x7f4ef414be90 2024-03-25T18:26:10.288 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.287+0000 7f4ef37fe640 1 -- 172.21.15.97:0/602066020 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3330625998 0 0) 0x7f4ef40af940 con 0x7f4ef40b1730 2024-03-25T18:26:10.288 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.287+0000 7f4ef37fe640 10 monclient(hunting): _negotiate 2024-03-25T18:26:10.288 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.287+0000 7f4ef37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:10.288 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.287+0000 7f4ef37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:10.288 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.287+0000 7f4ef37fe640 10 monclient(hunting): my global_id is 4668 2024-03-25T18:26:10.288 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.287+0000 7f4ef37fe640 1 -- 172.21.15.97:0/602066020 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ef40b0a70 con 0x7f4ef40b1730 2024-03-25T18:26:10.291 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.290+0000 7f4ef37fe640 1 -- 172.21.15.97:0/602066020 <== mon.2 v1:172.21.15.97:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1439559296 0 0) 0x7f4ee0003520 con 0x7f4ef40b4d60 2024-03-25T18:26:10.291 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.290+0000 7f4ef37fe640 1 -- 172.21.15.97:0/602066020 >> v1:172.21.15.98:6789/0 conn(0x7f4ef414be90 legacy=0x7f4ef414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:10.291 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.290+0000 7f4ef37fe640 1 -- 172.21.15.97:0/602066020 >> v1:172.21.15.97:6789/0 conn(0x7f4ef40b1730 legacy=0x7f4ef40b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:10.291 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.290+0000 7f4ef37fe640 10 monclient: _finish_hunting 0 2024-03-25T18:26:10.291 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.290+0000 7f4ef37fe640 1 monclient: found mon.noname-b 2024-03-25T18:26:10.291 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.290+0000 7f4ef37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:10.291 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.290+0000 7f4ef37fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.97:6790/0 2024-03-25T18:26:10.291 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.290+0000 7f4ef37fe640 1 -- 172.21.15.97:0/602066020 --> v1:172.21.15.97:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ef40b1030 con 0x7f4ef40b4d60 2024-03-25T18:26:10.291 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.290+0000 7f4ef37fe640 10 monclient: _finish_auth 0 2024-03-25T18:26:10.292 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.290+0000 7f4ef37fe640 10 monclient: _check_auth_tickets 2024-03-25T18:26:10.292 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.290+0000 7f4ef37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:10.292 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.290+0000 7f4ef37fe640 1 -- 172.21.15.97:0/602066020 <== mon.2 v1:172.21.15.97:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f4ee4001690 con 0x7f4ef40b4d60 2024-03-25T18:26:10.292 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.290+0000 7f4efa3c7640 5 monclient: authenticate success, global_id 4652 2024-03-25T18:26:10.292 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.290+0000 7f4efa3c7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:26:10.292 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.290+0000 7f4ef37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:10.292 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.290+0000 7f4ef37fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:26:10.292 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.290+0000 7f4ef37fe640 10 monclient: dump: 2024-03-25T18:26:10.292 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:26:10.292 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:26:10.292 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:26:10.292 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:26:10.292 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:10.292 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:26:10.293 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:26:10.293 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:26:10.293 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:26:10.293 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:26:10.293 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.290+0000 7f4efa3c7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:26:10.308 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.307+0000 7f4ef37fe640 1 -- 172.21.15.97:0/602066020 <== mon.2 v1:172.21.15.97:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4ee40017f0 con 0x7f4ef40b4d60 2024-03-25T18:26:10.308 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.307+0000 7f4ef37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:10.308 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.307+0000 7f4ef37fe640 1 -- 172.21.15.97:0/602066020 <== mon.2 v1:172.21.15.97:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f4ee4001c00 con 0x7f4ef40b4d60 2024-03-25T18:26:10.308 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.307+0000 7f4efa3c7640 10 monclient: get_monmap_and_config success 2024-03-25T18:26:10.309 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.307+0000 7f4efa3c7640 4 set_mon_vals no callback set 2024-03-25T18:26:10.309 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.307+0000 7f4ef37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:10.309 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.307+0000 7f4ef37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:26:10.309 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.307+0000 7f4ef37fe640 10 monclient: dump: 2024-03-25T18:26:10.309 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:26:10.309 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:26:10.309 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:26:10.309 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:26:10.309 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:10.309 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:26:10.309 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:26:10.310 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:26:10.310 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:26:10.310 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:26:10.312 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.308+0000 7f4efa3c7640 10 monclient: shutdown 2024-03-25T18:26:10.313 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.308+0000 7f4efa3c7640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:10.313 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.308+0000 7f4efa3c7640 1 -- 172.21.15.97:0/602066020 >> v1:172.21.15.97:6790/0 conn(0x7f4ef40b4d60 legacy=0x7f4ef413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:10.313 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.308+0000 7f4efa3c7640 1 -- 172.21.15.97:0/602066020 shutdown_connections 2024-03-25T18:26:10.313 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.308+0000 7f4efa3c7640 1 -- 172.21.15.97:0/602066020 >> 172.21.15.97:0/602066020 conn(0x7f4ef40617a0 msgr2=0x7f4ef4061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:10.313 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.308+0000 7f4efa3c7640 1 -- 172.21.15.97:0/602066020 shutdown_connections 2024-03-25T18:26:10.313 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.309+0000 7f4efa3c7640 1 -- 172.21.15.97:0/602066020 wait complete. 2024-03-25T18:26:10.313 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.309+0000 7f4efa3c7640 10 monclient: build_initial_monmap 2024-03-25T18:26:10.313 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.309+0000 7f4efa3c7640 10 monclient: monmap: 2024-03-25T18:26:10.313 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:26:10.313 INFO:teuthology.misc.health.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:26:10.314 INFO:teuthology.misc.health.smithi097.stderr:last_changed 0.000000 2024-03-25T18:26:10.314 INFO:teuthology.misc.health.smithi097.stderr:created 0.000000 2024-03-25T18:26:10.314 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:10.314 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 1 2024-03-25T18:26:10.314 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:26:10.314 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T18:26:10.314 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-c 2024-03-25T18:26:10.314 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:26:10.314 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.310+0000 7f4efa3c7640 1 librados: starting msgr at 2024-03-25T18:26:10.314 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.310+0000 7f4efa3c7640 1 librados: starting objecter 2024-03-25T18:26:10.314 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.310+0000 7f4efa3c7640 1 Processor -- start 2024-03-25T18:26:10.314 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.310+0000 7f4efa3c7640 1 -- start start 2024-03-25T18:26:10.314 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.310+0000 7f4efa3c7640 1 librados: setting wanted keys 2024-03-25T18:26:10.314 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.310+0000 7f4efa3c7640 1 librados: calling monclient init 2024-03-25T18:26:10.315 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.310+0000 7f4efa3c7640 10 monclient: init 2024-03-25T18:26:10.315 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.311+0000 7f4efa3c7640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:10.315 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.311+0000 7f4efa3c7640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:26:10.315 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.311+0000 7f4efa3c7640 10 monclient(hunting): picked mon.noname-b con 0x7f4ef40b1730 addr v1:172.21.15.98:6789/0 2024-03-25T18:26:10.315 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.311+0000 7f4efa3c7640 10 monclient(hunting): picked mon.noname-a con 0x7f4ef40b4d60 addr v1:172.21.15.97:6789/0 2024-03-25T18:26:10.315 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.311+0000 7f4efa3c7640 10 monclient(hunting): picked mon.noname-c con 0x7f4ef414be90 addr v1:172.21.15.97:6790/0 2024-03-25T18:26:10.315 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.311+0000 7f4efa3c7640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ef413b9a0 con 0x7f4ef40b4d60 2024-03-25T18:26:10.315 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.311+0000 7f4efa3c7640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ef41393e0 con 0x7f4ef40b1730 2024-03-25T18:26:10.315 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.311+0000 7f4efa3c7640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ef4143900 con 0x7f4ef414be90 2024-03-25T18:26:10.315 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.311+0000 7f4efa3c7640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:10.315 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.311+0000 7f4efa3c7640 10 monclient(hunting): authenticate will time out at 8182.586314s 2024-03-25T18:26:10.316 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.315+0000 7f4ef3fff640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f4ef40b1730 0x7f4ef4139900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.97:50006/0 (socket says 172.21.15.97:50006) 2024-03-25T18:26:10.316 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.315+0000 7f4ef3fff640 1 -- 172.21.15.97:0/902447125 learned_addr learned my addr 172.21.15.97:0/902447125 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:26:10.320 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.319+0000 7f4ef17fa640 10 client.?.objecter ms_handle_connect 0x7f4ef40b1730 2024-03-25T18:26:10.320 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.319+0000 7f4ef17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:10.323 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.322+0000 7f4ef17fa640 1 -- 172.21.15.97:0/902447125 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3316209291 0 0) 0x7f4ef41393e0 con 0x7f4ef40b1730 2024-03-25T18:26:10.323 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.322+0000 7f4ef17fa640 10 monclient(hunting): _negotiate 2024-03-25T18:26:10.323 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.322+0000 7f4ef17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:10.323 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.322+0000 7f4ef17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:10.323 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.322+0000 7f4ef17fa640 10 monclient(hunting): my global_id is 4675 2024-03-25T18:26:10.323 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.322+0000 7f4ef17fa640 1 -- 172.21.15.97:0/902447125 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ed4004eb0 con 0x7f4ef40b1730 2024-03-25T18:26:10.324 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.324+0000 7f4ef17fa640 10 client.?.objecter ms_handle_connect 0x7f4ef414be90 2024-03-25T18:26:10.324 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.324+0000 7f4ef17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:10.328 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.328+0000 7f4ef17fa640 10 client.?.objecter ms_handle_connect 0x7f4ef40b4d60 2024-03-25T18:26:10.328 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.328+0000 7f4ef17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:10.333 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.332+0000 7f4ef17fa640 1 -- 172.21.15.97:0/902447125 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1872501414 0 0) 0x7f4ed4004eb0 con 0x7f4ef40b1730 2024-03-25T18:26:10.333 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.332+0000 7f4ef17fa640 1 -- 172.21.15.97:0/902447125 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4ef41393e0 con 0x7f4ef40b1730 2024-03-25T18:26:10.333 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.332+0000 7f4ef17fa640 1 -- 172.21.15.97:0/902447125 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f4ee4001bc0 con 0x7f4ef40b1730 2024-03-25T18:26:10.333 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.332+0000 7f4ef17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:10.333 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.332+0000 7f4ef17fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:26:10.333 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.332+0000 7f4ef17fa640 10 monclient(hunting): dump: 2024-03-25T18:26:10.333 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:26:10.333 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:26:10.333 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:26:10.333 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:26:10.333 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:10.333 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:26:10.334 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:26:10.334 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:26:10.334 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:26:10.334 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:26:10.334 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.332+0000 7f4ef17fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:26:10.334 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.332+0000 7f4ef17fa640 1 -- 172.21.15.97:0/902447125 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3567049982 0 0) 0x7f4ef4143900 con 0x7f4ef414be90 2024-03-25T18:26:10.334 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.332+0000 7f4ef17fa640 10 monclient(hunting): _negotiate 2024-03-25T18:26:10.334 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.332+0000 7f4ef17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:10.334 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.332+0000 7f4ef17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:10.334 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.332+0000 7f4ef17fa640 10 monclient(hunting): my global_id is 4655 2024-03-25T18:26:10.334 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.332+0000 7f4ef17fa640 1 -- 172.21.15.97:0/902447125 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ed4004eb0 con 0x7f4ef414be90 2024-03-25T18:26:10.337 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.336+0000 7f4ef17fa640 1 -- 172.21.15.97:0/902447125 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3154891587 0 0) 0x7f4ef413b9a0 con 0x7f4ef40b4d60 2024-03-25T18:26:10.337 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.336+0000 7f4ef17fa640 10 monclient(hunting): _negotiate 2024-03-25T18:26:10.337 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.336+0000 7f4ef17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:10.337 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.336+0000 7f4ef17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:10.337 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.336+0000 7f4ef17fa640 10 monclient(hunting): my global_id is 4671 2024-03-25T18:26:10.337 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.336+0000 7f4ef17fa640 1 -- 172.21.15.97:0/902447125 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ef4143900 con 0x7f4ef40b4d60 2024-03-25T18:26:10.338 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.337+0000 7f4ef17fa640 1 -- 172.21.15.97:0/902447125 <== mon.1 v1:172.21.15.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3450093914 0 0) 0x7f4ef41393e0 con 0x7f4ef40b1730 2024-03-25T18:26:10.338 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.337+0000 7f4ef17fa640 1 -- 172.21.15.97:0/902447125 >> v1:172.21.15.97:6790/0 conn(0x7f4ef414be90 legacy=0x7f4ef41d8120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:10.339 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.337+0000 7f4ef17fa640 1 -- 172.21.15.97:0/902447125 >> v1:172.21.15.97:6789/0 conn(0x7f4ef40b4d60 legacy=0x7f4ef41d7a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:10.339 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.337+0000 7f4ef17fa640 10 monclient: _finish_hunting 0 2024-03-25T18:26:10.339 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.337+0000 7f4ef17fa640 1 monclient: found mon.b 2024-03-25T18:26:10.339 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.337+0000 7f4ef17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:10.339 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.338+0000 7f4ef17fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:26:10.339 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.338+0000 7f4ef17fa640 1 -- 172.21.15.97:0/902447125 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ef4143a90 con 0x7f4ef40b1730 2024-03-25T18:26:10.339 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.338+0000 7f4ef17fa640 10 monclient: _finish_auth 0 2024-03-25T18:26:10.339 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.338+0000 7f4ef17fa640 10 monclient: _check_auth_tickets 2024-03-25T18:26:10.339 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.338+0000 7f4ef17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:10.339 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.338+0000 7f4efa3c7640 5 monclient: authenticate success, global_id 4675 2024-03-25T18:26:10.339 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.338+0000 7f4efa3c7640 10 monclient: _renew_subs 2024-03-25T18:26:10.340 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.338+0000 7f4efa3c7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:26:10.340 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.338+0000 7f4efa3c7640 1 -- 172.21.15.97:0/902447125 --> v1:172.21.15.98:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ef413a010 con 0x7f4ef40b1730 2024-03-25T18:26:10.340 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.338+0000 7f4efa3c7640 10 client.4675.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:26:10.340 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.338+0000 7f4efa3c7640 10 monclient: _renew_subs 2024-03-25T18:26:10.340 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.338+0000 7f4efa3c7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:26:10.340 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.338+0000 7f4efa3c7640 1 -- 172.21.15.97:0/902447125 --> v1:172.21.15.98:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ef41e3470 con 0x7f4ef40b1730 2024-03-25T18:26:10.340 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.338+0000 7f4efa3c7640 1 librados: init done 2024-03-25T18:26:10.341 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.339+0000 7f4efa3c7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:10.341 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.339+0000 7f4efa3c7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:10.341 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.339+0000 7f4efa3c7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:26:10.341 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.339+0000 7f4efa3c7640 1 -- 172.21.15.97:0/902447125 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ef40627b0 con 0x7f4ef40b1730 2024-03-25T18:26:10.355 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.354+0000 7f4ef17fa640 1 -- 172.21.15.97:0/902447125 <== mon.1 v1:172.21.15.98:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4ee40011f0 con 0x7f4ef40b1730 2024-03-25T18:26:10.355 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.354+0000 7f4ef17fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:10.355 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.354+0000 7f4ef17fa640 1 -- 172.21.15.97:0/902447125 <== mon.1 v1:172.21.15.98:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f4ee4004c70 con 0x7f4ef40b1730 2024-03-25T18:26:10.355 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.354+0000 7f4ef17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:10.355 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.354+0000 7f4ef17fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:26:10.355 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.354+0000 7f4ef17fa640 10 monclient: dump: 2024-03-25T18:26:10.355 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:26:10.355 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:26:10.355 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:26:10.355 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:26:10.356 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:10.356 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:26:10.356 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:26:10.356 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:26:10.356 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:26:10.356 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:26:10.356 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.354+0000 7f4ef2ffd640 4 set_mon_vals no callback set 2024-03-25T18:26:10.357 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.355+0000 7f4ef17fa640 1 -- 172.21.15.97:0/902447125 <== mon.1 v1:172.21.15.98:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2377622137 0 0) 0x7f4ee4004e90 con 0x7f4ef40b1730 2024-03-25T18:26:10.357 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.356+0000 7f4eeb7fe640 10 monclient: get_auth_request con 0x7f4ed406de10 auth_method 0 2024-03-25T18:26:10.357 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.356+0000 7f4ef17fa640 10 client.4675.objecter ms_handle_connect 0x7f4ed406de10 2024-03-25T18:26:10.358 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.358+0000 7f4ef17fa640 1 -- 172.21.15.97:0/902447125 <== mon.1 v1:172.21.15.98:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 430711918 0 0) 0x7f4ee40549e0 con 0x7f4ef40b1730 2024-03-25T18:26:10.358 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.358+0000 7f4ef17fa640 10 client.4675.objecter ms_dispatch 0x7f4ef4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:26:10.359 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.358+0000 7f4ef17fa640 3 client.4675.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:26:10.359 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.358+0000 7f4ef17fa640 3 client.4675.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:26:10.359 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.358+0000 7f4ef17fa640 20 client.4675.objecter dump_active .. 0 homeless 2024-03-25T18:26:10.632 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.631+0000 7f4ef17fa640 1 -- 172.21.15.97:0/902447125 <== mon.1 v1:172.21.15.98:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4ee4050320 con 0x7f4ef40b1730 2024-03-25T18:26:10.632 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.631+0000 7f4ef17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:10.632 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.631+0000 7f4ef17fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:26:10.856 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.855+0000 7f4efa3c7640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:10.856 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.855+0000 7f4efa3c7640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:10.856 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.855+0000 7f4efa3c7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:26:10.856 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.855+0000 7f4efa3c7640 1 -- 172.21.15.97:0/902447125 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4ef4069f40 con 0x7f4ef40b1730 2024-03-25T18:26:10.873 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.872+0000 7f4ef17fa640 1 -- 172.21.15.97:0/902447125 <== mon.1 v1:172.21.15.98:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2169427412) 0x7f4ee4052420 con 0x7f4ef40b1730 2024-03-25T18:26:10.873 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.872+0000 7f4ef17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:10.873 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.872+0000 7f4ef17fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:26:10.873 INFO:teuthology.misc.health.smithi097.stdout: 2024-03-25T18:26:10.873 INFO:teuthology.misc.health.smithi097.stdout:{"epoch":10,"fsid":"4c3d472c-79b8-486b-9e3d-2738d4306754","created":"2024-03-25T18:16:14.737309+0000","modified":"2024-03-25T18:16:20.357492+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:20.357492+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cb5152b9-262f-45c3-94b8-18d0e81df558","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6daf8793-0838-473d-a7d6-9a0d2894c75d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"136f1493-3b9c-4f43-a2e7-641482d69dae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"397d538c-4f64-4daf-9e8b-662d09f91327","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4f1df08-663f-4bda-93f8-3878ab7c24b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c4b18d70-a3ce-4b15-9b70-eae5525bd761","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8dd4033a-de0e-4f8a-96dc-59f3c1f825b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0077722e-ffa6-4a3d-9891-929d9869b286","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:26:10.875 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.874+0000 7f4efa3c7640 10 librados: watch_flush enter 2024-03-25T18:26:10.876 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.874+0000 7f4efa3c7640 10 librados: watch_flush exit 2024-03-25T18:26:10.876 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.874+0000 7f4efa3c7640 20 client.4675.objecter shutdown clearing up homeless session... 2024-03-25T18:26:10.876 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.874+0000 7f4efa3c7640 10 client.4675.objecter successfully canceled tick 2024-03-25T18:26:10.876 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.874+0000 7f4efa3c7640 1 -- 172.21.15.97:0/902447125 >> v1:172.21.15.98:6800/800387573 conn(0x7f4ed406de10 legacy=0x7f4ed40702d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:10.876 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.874+0000 7f4efa3c7640 10 monclient: shutdown 2024-03-25T18:26:10.876 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.874+0000 7f4efa3c7640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:10.876 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.874+0000 7f4efa3c7640 1 -- 172.21.15.97:0/902447125 >> v1:172.21.15.98:6789/0 conn(0x7f4ef40b1730 legacy=0x7f4ef4139900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:10.876 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.875+0000 7f4efa3c7640 1 -- 172.21.15.97:0/902447125 shutdown_connections 2024-03-25T18:26:10.876 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.875+0000 7f4efa3c7640 1 -- 172.21.15.97:0/902447125 >> 172.21.15.97:0/902447125 conn(0x7f4ef40617a0 msgr2=0x7f4ef4061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:10.876 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.875+0000 7f4efa3c7640 1 -- 172.21.15.97:0/902447125 shutdown_connections 2024-03-25T18:26:10.876 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.875+0000 7f4efa3c7640 1 -- 172.21.15.97:0/902447125 wait complete. 2024-03-25T18:26:10.876 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:10.875+0000 7f4efa3c7640 1 librados: shutdown 2024-03-25T18:26:10.888 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:26:17.695 DEBUG:teuthology.orchestra.run.smithi097:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:26:17.811 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.808+0000 7fb04e657640 10 monclient: get_monmap_and_config 2024-03-25T18:26:17.811 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.808+0000 7fb04e657640 10 monclient: build_initial_monmap 2024-03-25T18:26:17.811 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.808+0000 7fb04e657640 10 monclient: monmap: 2024-03-25T18:26:17.811 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:26:17.811 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:26:17.811 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:26:17.809992+0000 2024-03-25T18:26:17.811 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:26:17.809992+0000 2024-03-25T18:26:17.811 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:17.811 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:26:17.812 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:26:17.812 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T18:26:17.812 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-b 2024-03-25T18:26:17.812 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:26:17.812 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.809+0000 7fb04e657640 1 Processor -- start 2024-03-25T18:26:17.812 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.809+0000 7fb04e657640 1 -- start start 2024-03-25T18:26:17.812 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.809+0000 7fb04e657640 10 monclient: init 2024-03-25T18:26:17.813 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.810+0000 7fb04e657640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:17.813 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.810+0000 7fb04e657640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:26:17.813 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.810+0000 7fb04e657640 10 monclient(hunting): picked mon.noname-c con 0x7fb0480a98f0 addr v1:172.21.15.98:6789/0 2024-03-25T18:26:17.813 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.810+0000 7fb04e657640 10 monclient(hunting): picked mon.noname-b con 0x7fb04813f710 addr v1:172.21.15.97:6790/0 2024-03-25T18:26:17.813 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.810+0000 7fb04e657640 10 monclient(hunting): picked mon.noname-a con 0x7fb0481430c0 addr v1:172.21.15.97:6789/0 2024-03-25T18:26:17.813 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.810+0000 7fb04e657640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb048142520 con 0x7fb0481430c0 2024-03-25T18:26:17.813 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.810+0000 7fb04e657640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0480b28a0 con 0x7fb0480a98f0 2024-03-25T18:26:17.813 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.810+0000 7fb04e657640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0480b2a30 con 0x7fb04813f710 2024-03-25T18:26:17.814 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.810+0000 7fb04e657640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:17.814 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.810+0000 7fb04e657640 10 monclient(hunting): authenticate will time out at 8190.086377s 2024-03-25T18:26:17.818 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.817+0000 7fb047fff640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fb0480a98f0 0x7fb0480a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.97:57268/0 (socket says 172.21.15.97:57268) 2024-03-25T18:26:17.818 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.817+0000 7fb047fff640 1 -- 172.21.15.97:0/1153073512 learned_addr learned my addr 172.21.15.97:0/1153073512 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:26:17.830 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.828+0000 7fb046ffd640 1 -- 172.21.15.97:0/1153073512 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1631691894 0 0) 0x7fb0480b28a0 con 0x7fb0480a98f0 2024-03-25T18:26:17.830 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.828+0000 7fb046ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:26:17.830 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.828+0000 7fb046ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:17.830 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.828+0000 7fb046ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:17.830 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.828+0000 7fb046ffd640 10 monclient(hunting): my global_id is 4678 2024-03-25T18:26:17.830 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.828+0000 7fb046ffd640 1 -- 172.21.15.97:0/1153073512 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb030003520 con 0x7fb0480a98f0 2024-03-25T18:26:17.832 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.831+0000 7fb046ffd640 1 -- 172.21.15.97:0/1153073512 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 617361006 0 0) 0x7fb048142520 con 0x7fb0481430c0 2024-03-25T18:26:17.833 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.831+0000 7fb046ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:26:17.833 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.831+0000 7fb046ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:17.833 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.831+0000 7fb046ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:17.833 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.831+0000 7fb046ffd640 10 monclient(hunting): my global_id is 4674 2024-03-25T18:26:17.833 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.831+0000 7fb046ffd640 1 -- 172.21.15.97:0/1153073512 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0480b28a0 con 0x7fb0481430c0 2024-03-25T18:26:17.843 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.841+0000 7fb046ffd640 1 -- 172.21.15.97:0/1153073512 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 19323833 0 0) 0x7fb0480b2a30 con 0x7fb04813f710 2024-03-25T18:26:17.843 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.841+0000 7fb046ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:26:17.843 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.841+0000 7fb046ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:17.843 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.841+0000 7fb046ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:17.843 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.841+0000 7fb046ffd640 10 monclient(hunting): my global_id is 4658 2024-03-25T18:26:17.843 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.841+0000 7fb046ffd640 1 -- 172.21.15.97:0/1153073512 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb048142520 con 0x7fb04813f710 2024-03-25T18:26:17.852 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.850+0000 7fb046ffd640 1 -- 172.21.15.97:0/1153073512 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1434439987 0 0) 0x7fb0480b28a0 con 0x7fb0481430c0 2024-03-25T18:26:17.852 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.850+0000 7fb046ffd640 1 -- 172.21.15.97:0/1153073512 >> v1:172.21.15.97:6790/0 conn(0x7fb04813f710 legacy=0x7fb0481429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:17.852 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.850+0000 7fb046ffd640 1 -- 172.21.15.97:0/1153073512 >> v1:172.21.15.98:6789/0 conn(0x7fb0480a98f0 legacy=0x7fb0480a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:17.852 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.850+0000 7fb046ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:26:17.852 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.850+0000 7fb046ffd640 1 monclient: found mon.noname-a 2024-03-25T18:26:17.852 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.850+0000 7fb046ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:17.852 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.850+0000 7fb046ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.97:6789/0 2024-03-25T18:26:17.852 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.850+0000 7fb046ffd640 1 -- 172.21.15.97:0/1153073512 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0480b44c0 con 0x7fb0481430c0 2024-03-25T18:26:17.853 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.850+0000 7fb046ffd640 10 monclient: _finish_auth 0 2024-03-25T18:26:17.853 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.850+0000 7fb046ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:26:17.853 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.850+0000 7fb046ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:17.853 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.850+0000 7fb046ffd640 1 -- 172.21.15.97:0/1153073512 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7fb0400019f0 con 0x7fb0481430c0 2024-03-25T18:26:17.853 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.850+0000 7fb046ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:17.853 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.850+0000 7fb046ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:26:17.853 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.850+0000 7fb046ffd640 10 monclient: dump: 2024-03-25T18:26:17.853 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:26:17.853 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:26:17.853 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:26:17.853 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:26:17.853 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:17.854 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:26:17.854 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:26:17.854 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:26:17.854 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:26:17.854 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:26:17.854 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.850+0000 7fb04e657640 5 monclient: authenticate success, global_id 4674 2024-03-25T18:26:17.854 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.850+0000 7fb04e657640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:26:17.862 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.861+0000 7fb046ffd640 1 -- 172.21.15.97:0/1153073512 <== mon.0 v1:172.21.15.97:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb040001b50 con 0x7fb0481430c0 2024-03-25T18:26:17.862 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.861+0000 7fb046ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:17.863 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.861+0000 7fb046ffd640 1 -- 172.21.15.97:0/1153073512 <== mon.0 v1:172.21.15.97:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7fb040001f60 con 0x7fb0481430c0 2024-03-25T18:26:17.863 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.861+0000 7fb046ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:17.863 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.861+0000 7fb046ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:26:17.863 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.861+0000 7fb046ffd640 10 monclient: dump: 2024-03-25T18:26:17.863 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:26:17.863 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:26:17.863 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:26:17.863 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:26:17.863 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:17.863 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:26:17.863 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:26:17.863 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:26:17.863 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:26:17.863 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:26:17.864 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.861+0000 7fb04e657640 10 monclient: get_monmap_and_config success 2024-03-25T18:26:17.864 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.861+0000 7fb04e657640 4 set_mon_vals no callback set 2024-03-25T18:26:17.865 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.862+0000 7fb04e657640 10 monclient: shutdown 2024-03-25T18:26:17.865 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.862+0000 7fb04e657640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:17.865 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.862+0000 7fb04e657640 1 -- 172.21.15.97:0/1153073512 >> v1:172.21.15.97:6789/0 conn(0x7fb0481430c0 legacy=0x7fb04814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:17.865 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.862+0000 7fb04e657640 1 -- 172.21.15.97:0/1153073512 shutdown_connections 2024-03-25T18:26:17.865 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.862+0000 7fb04e657640 1 -- 172.21.15.97:0/1153073512 >> 172.21.15.97:0/1153073512 conn(0x7fb0480b7ea0 msgr2=0x7fb0480b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:17.865 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.862+0000 7fb04e657640 1 -- 172.21.15.97:0/1153073512 shutdown_connections 2024-03-25T18:26:17.865 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.862+0000 7fb04e657640 1 -- 172.21.15.97:0/1153073512 wait complete. 2024-03-25T18:26:17.866 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.862+0000 7fb04e657640 10 monclient: build_initial_monmap 2024-03-25T18:26:17.866 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.862+0000 7fb04e657640 10 monclient: monmap: 2024-03-25T18:26:17.866 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:26:17.866 INFO:teuthology.misc.health.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:26:17.866 INFO:teuthology.misc.health.smithi097.stderr:last_changed 0.000000 2024-03-25T18:26:17.866 INFO:teuthology.misc.health.smithi097.stderr:created 0.000000 2024-03-25T18:26:17.866 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:17.866 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 1 2024-03-25T18:26:17.866 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:26:17.866 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T18:26:17.866 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-c 2024-03-25T18:26:17.866 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:26:17.866 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.863+0000 7fb04e657640 1 librados: starting msgr at 2024-03-25T18:26:17.866 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.863+0000 7fb04e657640 1 librados: starting objecter 2024-03-25T18:26:17.867 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.863+0000 7fb04e657640 1 Processor -- start 2024-03-25T18:26:17.867 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.863+0000 7fb04e657640 1 -- start start 2024-03-25T18:26:17.867 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.863+0000 7fb04e657640 1 librados: setting wanted keys 2024-03-25T18:26:17.867 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.863+0000 7fb04e657640 1 librados: calling monclient init 2024-03-25T18:26:17.867 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.863+0000 7fb04e657640 10 monclient: init 2024-03-25T18:26:17.868 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.864+0000 7fb04e657640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:17.868 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.864+0000 7fb04e657640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:26:17.868 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.864+0000 7fb04e657640 10 monclient(hunting): picked mon.noname-c con 0x7fb0480a98f0 addr v1:172.21.15.97:6790/0 2024-03-25T18:26:17.868 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.864+0000 7fb04e657640 10 monclient(hunting): picked mon.noname-b con 0x7fb04813f710 addr v1:172.21.15.98:6789/0 2024-03-25T18:26:17.868 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.864+0000 7fb04e657640 10 monclient(hunting): picked mon.noname-a con 0x7fb0481430c0 addr v1:172.21.15.97:6789/0 2024-03-25T18:26:17.868 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.864+0000 7fb04e657640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0481d48f0 con 0x7fb0481430c0 2024-03-25T18:26:17.868 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.864+0000 7fb04e657640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0481397d0 con 0x7fb04813f710 2024-03-25T18:26:17.868 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.864+0000 7fb04e657640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0481d4b50 con 0x7fb0480a98f0 2024-03-25T18:26:17.868 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.864+0000 7fb04e657640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:17.868 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.864+0000 7fb04e657640 10 monclient(hunting): authenticate will time out at 8190.140306s 2024-03-25T18:26:17.876 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.874+0000 7fb0477fe640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fb04813f710 0x7fb0480aff00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.97:57308/0 (socket says 172.21.15.97:57308) 2024-03-25T18:26:17.876 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.874+0000 7fb0477fe640 1 -- 172.21.15.97:0/134532195 learned_addr learned my addr 172.21.15.97:0/134532195 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:26:17.878 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.876+0000 7fb044ff9640 10 client.?.objecter ms_handle_connect 0x7fb04813f710 2024-03-25T18:26:17.878 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.876+0000 7fb044ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:17.880 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.878+0000 7fb044ff9640 10 client.?.objecter ms_handle_connect 0x7fb0480a98f0 2024-03-25T18:26:17.880 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.878+0000 7fb044ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:17.883 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.882+0000 7fb044ff9640 10 client.?.objecter ms_handle_connect 0x7fb0481430c0 2024-03-25T18:26:17.883 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.882+0000 7fb044ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:17.884 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.883+0000 7fb044ff9640 1 -- 172.21.15.97:0/134532195 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3199960364 0 0) 0x7fb0481397d0 con 0x7fb04813f710 2024-03-25T18:26:17.884 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.883+0000 7fb044ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:26:17.884 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.883+0000 7fb044ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:17.884 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.883+0000 7fb044ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:17.885 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.883+0000 7fb044ff9640 10 monclient(hunting): my global_id is 4681 2024-03-25T18:26:17.885 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.883+0000 7fb044ff9640 1 -- 172.21.15.97:0/134532195 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb034004eb0 con 0x7fb04813f710 2024-03-25T18:26:17.891 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.889+0000 7fb044ff9640 1 -- 172.21.15.97:0/134532195 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2148140271 0 0) 0x7fb0481d4b50 con 0x7fb0480a98f0 2024-03-25T18:26:17.891 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.889+0000 7fb044ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:26:17.891 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.889+0000 7fb044ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:17.891 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.889+0000 7fb044ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:17.891 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.889+0000 7fb044ff9640 10 monclient(hunting): my global_id is 4661 2024-03-25T18:26:17.891 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.889+0000 7fb044ff9640 1 -- 172.21.15.97:0/134532195 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0481397d0 con 0x7fb0480a98f0 2024-03-25T18:26:17.894 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.893+0000 7fb044ff9640 1 -- 172.21.15.97:0/134532195 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3596286060 0 0) 0x7fb0481d48f0 con 0x7fb0481430c0 2024-03-25T18:26:17.895 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.893+0000 7fb044ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:26:17.895 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.893+0000 7fb044ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:17.895 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.893+0000 7fb044ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:17.895 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.893+0000 7fb044ff9640 10 monclient(hunting): my global_id is 4677 2024-03-25T18:26:17.895 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.893+0000 7fb044ff9640 1 -- 172.21.15.97:0/134532195 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0481d4b50 con 0x7fb0481430c0 2024-03-25T18:26:17.897 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.895+0000 7fb044ff9640 1 -- 172.21.15.97:0/134532195 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2514768706 0 0) 0x7fb034004eb0 con 0x7fb04813f710 2024-03-25T18:26:17.897 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.895+0000 7fb044ff9640 1 -- 172.21.15.97:0/134532195 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb0481d48f0 con 0x7fb04813f710 2024-03-25T18:26:17.897 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.895+0000 7fb044ff9640 1 -- 172.21.15.97:0/134532195 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7fb03c001a10 con 0x7fb04813f710 2024-03-25T18:26:17.897 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.895+0000 7fb044ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:17.897 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.895+0000 7fb044ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:26:17.897 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.895+0000 7fb044ff9640 10 monclient(hunting): dump: 2024-03-25T18:26:17.897 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:26:17.897 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:26:17.897 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:26:17.897 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:26:17.897 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:17.898 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:26:17.898 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:26:17.898 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:26:17.898 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:26:17.898 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:26:17.898 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.895+0000 7fb044ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:26:17.908 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.906+0000 7fb044ff9640 1 -- 172.21.15.97:0/134532195 <== mon.2 v1:172.21.15.97:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2852246569 0 0) 0x7fb0481397d0 con 0x7fb0480a98f0 2024-03-25T18:26:17.908 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.906+0000 7fb044ff9640 1 -- 172.21.15.97:0/134532195 --> v1:172.21.15.97:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb034004eb0 con 0x7fb0480a98f0 2024-03-25T18:26:17.908 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.906+0000 7fb044ff9640 1 -- 172.21.15.97:0/134532195 <== mon.2 v1:172.21.15.97:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7fb038001950 con 0x7fb0480a98f0 2024-03-25T18:26:17.908 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.906+0000 7fb044ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:17.908 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.906+0000 7fb044ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:26:17.908 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.906+0000 7fb044ff9640 10 monclient(hunting): dump: 2024-03-25T18:26:17.908 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:26:17.909 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:26:17.909 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:26:17.909 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:26:17.909 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:17.909 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:26:17.909 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:26:17.909 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:26:17.909 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:26:17.909 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:26:17.910 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.907+0000 7fb044ff9640 1 -- 172.21.15.97:0/134532195 <== mon.1 v1:172.21.15.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3985828183 0 0) 0x7fb0481d48f0 con 0x7fb04813f710 2024-03-25T18:26:17.910 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.907+0000 7fb044ff9640 1 -- 172.21.15.97:0/134532195 >> v1:172.21.15.97:6790/0 conn(0x7fb0480a98f0 legacy=0x7fb0481d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:17.911 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.907+0000 7fb044ff9640 1 -- 172.21.15.97:0/134532195 >> v1:172.21.15.97:6789/0 conn(0x7fb0481430c0 legacy=0x7fb0481d7380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:17.911 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.907+0000 7fb044ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:26:17.911 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.907+0000 7fb044ff9640 1 monclient: found mon.b 2024-03-25T18:26:17.911 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.907+0000 7fb044ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:17.911 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.907+0000 7fb044ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:26:17.911 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.907+0000 7fb044ff9640 1 -- 172.21.15.97:0/134532195 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0480b16b0 con 0x7fb04813f710 2024-03-25T18:26:17.911 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.907+0000 7fb044ff9640 10 monclient: _finish_auth 0 2024-03-25T18:26:17.911 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.907+0000 7fb044ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:26:17.911 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.907+0000 7fb044ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:17.911 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.907+0000 7fb04e657640 5 monclient: authenticate success, global_id 4681 2024-03-25T18:26:17.912 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.907+0000 7fb04e657640 10 monclient: _renew_subs 2024-03-25T18:26:17.912 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.907+0000 7fb04e657640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:26:17.912 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.907+0000 7fb04e657640 1 -- 172.21.15.97:0/134532195 --> v1:172.21.15.98:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0480b0c10 con 0x7fb04813f710 2024-03-25T18:26:17.912 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.907+0000 7fb04e657640 10 client.4681.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:26:17.912 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.907+0000 7fb04e657640 10 monclient: _renew_subs 2024-03-25T18:26:17.912 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.907+0000 7fb04e657640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:26:17.912 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.907+0000 7fb04e657640 1 -- 172.21.15.97:0/134532195 --> v1:172.21.15.98:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0480b0fb0 con 0x7fb04813f710 2024-03-25T18:26:17.912 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.907+0000 7fb04e657640 1 librados: init done 2024-03-25T18:26:17.912 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.908+0000 7fb04e657640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:17.912 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.908+0000 7fb04e657640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:17.912 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.908+0000 7fb04e657640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:26:17.912 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:17.908+0000 7fb04e657640 1 -- 172.21.15.97:0/134532195 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0481d4ce0 con 0x7fb04813f710 2024-03-25T18:26:18.194 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:18.192+0000 7fb044ff9640 1 -- 172.21.15.97:0/134532195 <== mon.1 v1:172.21.15.98:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb03c001c70 con 0x7fb04813f710 2024-03-25T18:26:18.194 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:18.192+0000 7fb044ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:18.194 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:18.192+0000 7fb044ff9640 1 -- 172.21.15.97:0/134532195 <== mon.1 v1:172.21.15.98:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7fb03c003a60 con 0x7fb04813f710 2024-03-25T18:26:18.194 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:18.192+0000 7fb044ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:18.194 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:18.192+0000 7fb044ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:26:18.194 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:18.192+0000 7fb044ff9640 10 monclient: dump: 2024-03-25T18:26:18.194 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:26:18.194 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:26:18.194 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:26:18.195 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:26:18.195 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:18.195 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:26:18.195 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:26:18.195 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:26:18.195 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:26:18.195 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:26:18.195 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:18.192+0000 7fb0467fc640 4 set_mon_vals no callback set 2024-03-25T18:26:18.196 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:18.193+0000 7fb044ff9640 1 -- 172.21.15.97:0/134532195 <== mon.1 v1:172.21.15.98:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2377622137 0 0) 0x7fb03c003c40 con 0x7fb04813f710 2024-03-25T18:26:18.196 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:18.193+0000 7fb047fff640 10 monclient: get_auth_request con 0x7fb03406df70 auth_method 0 2024-03-25T18:26:18.196 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:18.193+0000 7fb044ff9640 10 client.4681.objecter ms_handle_connect 0x7fb03406df70 2024-03-25T18:26:18.199 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:18.197+0000 7fb044ff9640 1 -- 172.21.15.97:0/134532195 <== mon.1 v1:172.21.15.98:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 430711918 0 0) 0x7fb03c054890 con 0x7fb04813f710 2024-03-25T18:26:18.199 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:18.198+0000 7fb044ff9640 10 client.4681.objecter ms_dispatch 0x7fb048000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:26:18.199 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:18.198+0000 7fb044ff9640 3 client.4681.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:26:18.200 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:18.198+0000 7fb044ff9640 3 client.4681.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:26:18.200 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:18.198+0000 7fb044ff9640 20 client.4681.objecter dump_active .. 0 homeless 2024-03-25T18:26:18.200 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:18.198+0000 7fb044ff9640 1 -- 172.21.15.97:0/134532195 <== mon.1 v1:172.21.15.98:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb03c0503a0 con 0x7fb04813f710 2024-03-25T18:26:18.200 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:18.198+0000 7fb044ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:18.200 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:18.198+0000 7fb044ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:26:18.433 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:18.431+0000 7fb04e657640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:18.433 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:18.431+0000 7fb04e657640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:18.433 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:18.431+0000 7fb04e657640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:26:18.433 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:18.431+0000 7fb04e657640 1 -- 172.21.15.97:0/134532195 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb0480627b0 con 0x7fb04813f710 2024-03-25T18:26:18.477 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:18.475+0000 7fb044ff9640 1 -- 172.21.15.97:0/134532195 <== mon.1 v1:172.21.15.98:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2169427412) 0x7fb03c0522d0 con 0x7fb04813f710 2024-03-25T18:26:18.477 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:18.475+0000 7fb044ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:18.477 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:18.475+0000 7fb044ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:26:18.477 INFO:teuthology.misc.health.smithi097.stdout: 2024-03-25T18:26:18.477 INFO:teuthology.misc.health.smithi097.stdout:{"epoch":10,"fsid":"4c3d472c-79b8-486b-9e3d-2738d4306754","created":"2024-03-25T18:16:14.737309+0000","modified":"2024-03-25T18:16:20.357492+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:20.357492+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cb5152b9-262f-45c3-94b8-18d0e81df558","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6daf8793-0838-473d-a7d6-9a0d2894c75d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"136f1493-3b9c-4f43-a2e7-641482d69dae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"397d538c-4f64-4daf-9e8b-662d09f91327","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4f1df08-663f-4bda-93f8-3878ab7c24b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c4b18d70-a3ce-4b15-9b70-eae5525bd761","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8dd4033a-de0e-4f8a-96dc-59f3c1f825b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0077722e-ffa6-4a3d-9891-929d9869b286","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:26:18.481 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:18.478+0000 7fb04e657640 10 librados: watch_flush enter 2024-03-25T18:26:18.481 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:18.478+0000 7fb04e657640 10 librados: watch_flush exit 2024-03-25T18:26:18.481 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:18.478+0000 7fb04e657640 20 client.4681.objecter shutdown clearing up homeless session... 2024-03-25T18:26:18.481 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:18.478+0000 7fb04e657640 10 client.4681.objecter successfully canceled tick 2024-03-25T18:26:18.481 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:18.479+0000 7fb04e657640 1 -- 172.21.15.97:0/134532195 >> v1:172.21.15.98:6800/800387573 conn(0x7fb03406df70 legacy=0x7fb034070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:18.481 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:18.479+0000 7fb04e657640 10 monclient: shutdown 2024-03-25T18:26:18.481 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:18.479+0000 7fb04e657640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:18.481 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:18.479+0000 7fb04e657640 1 -- 172.21.15.97:0/134532195 >> v1:172.21.15.98:6789/0 conn(0x7fb04813f710 legacy=0x7fb0480aff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:18.481 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:18.479+0000 7fb04e657640 1 -- 172.21.15.97:0/134532195 shutdown_connections 2024-03-25T18:26:18.481 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:18.479+0000 7fb04e657640 1 -- 172.21.15.97:0/134532195 >> 172.21.15.97:0/134532195 conn(0x7fb0480b7ea0 msgr2=0x7fb048074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:18.482 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:18.479+0000 7fb04e657640 1 -- 172.21.15.97:0/134532195 shutdown_connections 2024-03-25T18:26:18.482 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:18.479+0000 7fb04e657640 1 -- 172.21.15.97:0/134532195 wait complete. 2024-03-25T18:26:18.482 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:18.479+0000 7fb04e657640 1 librados: shutdown 2024-03-25T18:26:18.492 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:26:19.000 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:10:21.038 34931== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:19.010 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:10:21.048 34931== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:19.039 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:10:21.077 34931== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:19.058 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:10:21.097 34935== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:19.065 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:10:21.103 34935== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:19.075 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:10:21.118 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:19.084 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:10:21.127 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:19.099 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:10:21.138 34935== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:19.108 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:10:21.151 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:25.300 DEBUG:teuthology.orchestra.run.smithi097:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:26:25.420 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.419+0000 7f48c29e2640 10 monclient: get_monmap_and_config 2024-03-25T18:26:25.421 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.419+0000 7f48c29e2640 10 monclient: build_initial_monmap 2024-03-25T18:26:25.421 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.419+0000 7f48c29e2640 10 monclient: monmap: 2024-03-25T18:26:25.421 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:26:25.421 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:26:25.421 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:26:25.420381+0000 2024-03-25T18:26:25.421 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:26:25.420381+0000 2024-03-25T18:26:25.421 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:25.421 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:26:25.421 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:26:25.421 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T18:26:25.421 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-b 2024-03-25T18:26:25.421 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:26:25.422 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.419+0000 7f48c29e2640 1 Processor -- start 2024-03-25T18:26:25.423 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.419+0000 7f48c29e2640 1 -- start start 2024-03-25T18:26:25.423 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.419+0000 7f48c29e2640 10 monclient: init 2024-03-25T18:26:25.423 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.420+0000 7f48c29e2640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:25.423 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.420+0000 7f48c29e2640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:26:25.423 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.420+0000 7f48c29e2640 10 monclient(hunting): picked mon.noname-b con 0x7f48bc0a98f0 addr v1:172.21.15.97:6790/0 2024-03-25T18:26:25.423 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.420+0000 7f48c29e2640 10 monclient(hunting): picked mon.noname-c con 0x7f48bc13f710 addr v1:172.21.15.98:6789/0 2024-03-25T18:26:25.423 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.420+0000 7f48c29e2640 10 monclient(hunting): picked mon.noname-a con 0x7f48bc1430c0 addr v1:172.21.15.97:6789/0 2024-03-25T18:26:25.423 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.420+0000 7f48c29e2640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48bc142520 con 0x7f48bc1430c0 2024-03-25T18:26:25.423 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.420+0000 7f48c29e2640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48bc0b28a0 con 0x7f48bc13f710 2024-03-25T18:26:25.423 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.420+0000 7f48c29e2640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48bc0b2a30 con 0x7f48bc0a98f0 2024-03-25T18:26:25.423 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.420+0000 7f48c29e2640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:25.424 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.420+0000 7f48c29e2640 10 monclient(hunting): authenticate will time out at 8197.696469s 2024-03-25T18:26:25.425 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.424+0000 7f48bb7fe640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f48bc13f710 0x7f48bc1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.97:50606/0 (socket says 172.21.15.97:50606) 2024-03-25T18:26:25.425 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.424+0000 7f48bb7fe640 1 -- 172.21.15.97:0/123310349 learned_addr learned my addr 172.21.15.97:0/123310349 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:26:25.435 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.433+0000 7f48baffd640 1 -- 172.21.15.97:0/123310349 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2076938735 0 0) 0x7f48bc0b28a0 con 0x7f48bc13f710 2024-03-25T18:26:25.435 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.433+0000 7f48baffd640 10 monclient(hunting): _negotiate 2024-03-25T18:26:25.435 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.433+0000 7f48baffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:25.435 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.433+0000 7f48baffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:25.436 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.433+0000 7f48baffd640 10 monclient(hunting): my global_id is 4684 2024-03-25T18:26:25.436 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.433+0000 7f48baffd640 1 -- 172.21.15.97:0/123310349 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48a4003520 con 0x7f48bc13f710 2024-03-25T18:26:25.439 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.437+0000 7f48baffd640 1 -- 172.21.15.97:0/123310349 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 305996432 0 0) 0x7f48a4003520 con 0x7f48bc13f710 2024-03-25T18:26:25.439 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.437+0000 7f48baffd640 1 -- 172.21.15.97:0/123310349 >> v1:172.21.15.97:6790/0 conn(0x7f48bc0a98f0 legacy=0x7f48bc0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:26:25.439 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.437+0000 7f48baffd640 1 -- 172.21.15.97:0/123310349 >> v1:172.21.15.97:6789/0 conn(0x7f48bc1430c0 legacy=0x7f48bc14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:26:25.439 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.437+0000 7f48baffd640 10 monclient: _finish_hunting 0 2024-03-25T18:26:25.439 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.437+0000 7f48baffd640 1 monclient: found mon.noname-c 2024-03-25T18:26:25.440 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.437+0000 7f48baffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:25.440 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.437+0000 7f48baffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.98:6789/0 2024-03-25T18:26:25.440 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.437+0000 7f48baffd640 1 -- 172.21.15.97:0/123310349 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48bc0b44c0 con 0x7f48bc13f710 2024-03-25T18:26:25.440 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.437+0000 7f48baffd640 10 monclient: _finish_auth 0 2024-03-25T18:26:25.440 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.437+0000 7f48baffd640 10 monclient: _check_auth_tickets 2024-03-25T18:26:25.440 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.437+0000 7f48baffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:25.440 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.437+0000 7f48c29e2640 5 monclient: authenticate success, global_id 4684 2024-03-25T18:26:25.440 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.437+0000 7f48c29e2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:26:25.442 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.440+0000 7f48baffd640 1 -- 172.21.15.97:0/123310349 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f48b0001690 con 0x7f48bc13f710 2024-03-25T18:26:25.442 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.440+0000 7f48baffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:25.442 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.440+0000 7f48baffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:26:25.442 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.440+0000 7f48baffd640 10 monclient: dump: 2024-03-25T18:26:25.442 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:26:25.442 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:26:25.442 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:26:25.443 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:26:25.443 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:25.443 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:26:25.443 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:26:25.443 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:26:25.443 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:26:25.443 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:26:25.443 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.440+0000 7f48baffd640 1 -- 172.21.15.97:0/123310349 <== mon.1 v1:172.21.15.98:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f48b0001960 con 0x7f48bc13f710 2024-03-25T18:26:25.443 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.440+0000 7f48c29e2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:26:25.443 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.440+0000 7f48baffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:25.443 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.440+0000 7f48c29e2640 10 monclient: get_monmap_and_config success 2024-03-25T18:26:25.443 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.440+0000 7f48c29e2640 4 set_mon_vals no callback set 2024-03-25T18:26:25.443 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.440+0000 7f48baffd640 1 -- 172.21.15.97:0/123310349 <== mon.1 v1:172.21.15.98:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f48b0001cf0 con 0x7f48bc13f710 2024-03-25T18:26:25.443 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.440+0000 7f48baffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:25.443 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.440+0000 7f48baffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:26:25.444 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.440+0000 7f48baffd640 10 monclient: dump: 2024-03-25T18:26:25.444 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:26:25.444 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:26:25.444 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:26:25.444 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:26:25.444 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:25.444 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:26:25.444 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:26:25.444 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:26:25.444 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:26:25.444 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:26:25.446 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.441+0000 7f48c29e2640 10 monclient: shutdown 2024-03-25T18:26:25.446 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.441+0000 7f48c29e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:25.446 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.441+0000 7f48c29e2640 1 -- 172.21.15.97:0/123310349 >> v1:172.21.15.98:6789/0 conn(0x7f48bc13f710 legacy=0x7f48bc1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:25.447 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.441+0000 7f48c29e2640 1 -- 172.21.15.97:0/123310349 shutdown_connections 2024-03-25T18:26:25.447 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.441+0000 7f48c29e2640 1 -- 172.21.15.97:0/123310349 >> 172.21.15.97:0/123310349 conn(0x7f48bc0b7ea0 msgr2=0x7f48bc0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:25.447 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.441+0000 7f48c29e2640 1 -- 172.21.15.97:0/123310349 shutdown_connections 2024-03-25T18:26:25.447 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.441+0000 7f48c29e2640 1 -- 172.21.15.97:0/123310349 wait complete. 2024-03-25T18:26:25.447 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.442+0000 7f48c29e2640 10 monclient: build_initial_monmap 2024-03-25T18:26:25.447 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.442+0000 7f48c29e2640 10 monclient: monmap: 2024-03-25T18:26:25.447 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:26:25.447 INFO:teuthology.misc.health.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:26:25.447 INFO:teuthology.misc.health.smithi097.stderr:last_changed 0.000000 2024-03-25T18:26:25.447 INFO:teuthology.misc.health.smithi097.stderr:created 0.000000 2024-03-25T18:26:25.447 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:25.447 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 1 2024-03-25T18:26:25.447 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:26:25.447 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T18:26:25.448 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-c 2024-03-25T18:26:25.448 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:26:25.448 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.442+0000 7f48c29e2640 1 librados: starting msgr at 2024-03-25T18:26:25.448 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.442+0000 7f48c29e2640 1 librados: starting objecter 2024-03-25T18:26:25.448 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.442+0000 7f48c29e2640 1 Processor -- start 2024-03-25T18:26:25.448 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.443+0000 7f48c29e2640 1 -- start start 2024-03-25T18:26:25.448 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.443+0000 7f48c29e2640 1 librados: setting wanted keys 2024-03-25T18:26:25.448 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.443+0000 7f48c29e2640 1 librados: calling monclient init 2024-03-25T18:26:25.448 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.443+0000 7f48c29e2640 10 monclient: init 2024-03-25T18:26:25.448 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.443+0000 7f48c29e2640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:25.448 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.443+0000 7f48c29e2640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:26:25.448 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.443+0000 7f48c29e2640 10 monclient(hunting): picked mon.noname-a con 0x7f48bc0a98f0 addr v1:172.21.15.97:6789/0 2024-03-25T18:26:25.448 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.443+0000 7f48c29e2640 10 monclient(hunting): picked mon.noname-c con 0x7f48bc13f710 addr v1:172.21.15.97:6790/0 2024-03-25T18:26:25.448 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.443+0000 7f48c29e2640 10 monclient(hunting): picked mon.noname-b con 0x7f48bc1430c0 addr v1:172.21.15.98:6789/0 2024-03-25T18:26:25.449 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.443+0000 7f48c29e2640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48bc0b1630 con 0x7f48bc0a98f0 2024-03-25T18:26:25.449 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.443+0000 7f48c29e2640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48bc0b0820 con 0x7f48bc1430c0 2024-03-25T18:26:25.449 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.443+0000 7f48c29e2640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48bc0b0b00 con 0x7f48bc13f710 2024-03-25T18:26:25.449 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.443+0000 7f48c29e2640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:25.449 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.443+0000 7f48c29e2640 10 monclient(hunting): authenticate will time out at 8197.719835s 2024-03-25T18:26:25.458 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.457+0000 7f48bb7fe640 1 --1- >> v1:172.21.15.97:6790/0 conn(0x7f48bc13f710 0x7f48bc0acb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.97:6790/0 says I am v1:172.21.15.97:54384/0 (socket says 172.21.15.97:54384) 2024-03-25T18:26:25.458 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.457+0000 7f48bb7fe640 1 -- 172.21.15.97:0/399765487 learned_addr learned my addr 172.21.15.97:0/399765487 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:26:25.459 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.457+0000 7f48b8ff9640 10 client.?.objecter ms_handle_connect 0x7f48bc13f710 2024-03-25T18:26:25.459 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.457+0000 7f48b8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:25.466 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.465+0000 7f48b8ff9640 1 -- 172.21.15.97:0/399765487 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1001419307 0 0) 0x7f48bc0b0b00 con 0x7f48bc13f710 2024-03-25T18:26:25.467 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.465+0000 7f48b8ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:26:25.467 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.465+0000 7f48b8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:25.467 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.465+0000 7f48b8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:25.467 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.465+0000 7f48b8ff9640 10 monclient(hunting): my global_id is 4664 2024-03-25T18:26:25.467 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.465+0000 7f48b8ff9640 1 -- 172.21.15.97:0/399765487 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48a8004eb0 con 0x7f48bc13f710 2024-03-25T18:26:25.467 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.465+0000 7f48b8ff9640 10 client.?.objecter ms_handle_connect 0x7f48bc1430c0 2024-03-25T18:26:25.467 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.465+0000 7f48b8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:25.474 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.473+0000 7f48b8ff9640 10 client.?.objecter ms_handle_connect 0x7f48bc0a98f0 2024-03-25T18:26:25.475 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.473+0000 7f48b8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:25.476 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.474+0000 7f48b8ff9640 1 -- 172.21.15.97:0/399765487 <== mon.2 v1:172.21.15.97:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2771718930 0 0) 0x7f48a8004eb0 con 0x7f48bc13f710 2024-03-25T18:26:25.476 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.474+0000 7f48b8ff9640 1 -- 172.21.15.97:0/399765487 --> v1:172.21.15.97:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f48bc0b0b00 con 0x7f48bc13f710 2024-03-25T18:26:25.477 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.474+0000 7f48b8ff9640 1 -- 172.21.15.97:0/399765487 <== mon.2 v1:172.21.15.97:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f48b0001650 con 0x7f48bc13f710 2024-03-25T18:26:25.477 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.475+0000 7f48b8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:25.477 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.475+0000 7f48b8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:26:25.477 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.475+0000 7f48b8ff9640 10 monclient(hunting): dump: 2024-03-25T18:26:25.477 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:26:25.477 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:26:25.477 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:26:25.477 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:26:25.477 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:25.477 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:26:25.477 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:26:25.477 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:26:25.477 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:26:25.477 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:26:25.478 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.475+0000 7f48b8ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:26:25.484 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.483+0000 7f48b8ff9640 1 -- 172.21.15.97:0/399765487 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 247899512 0 0) 0x7f48bc0b0820 con 0x7f48bc1430c0 2024-03-25T18:26:25.485 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.483+0000 7f48b8ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:26:25.485 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.483+0000 7f48b8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:25.485 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.483+0000 7f48b8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:25.485 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.483+0000 7f48b8ff9640 10 monclient(hunting): my global_id is 4687 2024-03-25T18:26:25.485 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.483+0000 7f48b8ff9640 1 -- 172.21.15.97:0/399765487 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48a8004eb0 con 0x7f48bc1430c0 2024-03-25T18:26:25.491 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.490+0000 7f48b8ff9640 1 -- 172.21.15.97:0/399765487 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2232425721 0 0) 0x7f48bc0b1630 con 0x7f48bc0a98f0 2024-03-25T18:26:25.492 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.490+0000 7f48b8ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:26:25.492 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.490+0000 7f48b8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:25.492 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.490+0000 7f48b8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:25.492 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.490+0000 7f48b8ff9640 10 monclient(hunting): my global_id is 4680 2024-03-25T18:26:25.492 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.490+0000 7f48b8ff9640 1 -- 172.21.15.97:0/399765487 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48bc0b0820 con 0x7f48bc0a98f0 2024-03-25T18:26:25.502 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.499+0000 7f48b8ff9640 1 -- 172.21.15.97:0/399765487 <== mon.2 v1:172.21.15.97:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3803181934 0 0) 0x7f48bc0b0b00 con 0x7f48bc13f710 2024-03-25T18:26:25.502 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.499+0000 7f48b8ff9640 1 -- 172.21.15.97:0/399765487 >> v1:172.21.15.98:6789/0 conn(0x7f48bc1430c0 legacy=0x7f48bc0b0f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:25.502 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.499+0000 7f48b8ff9640 1 -- 172.21.15.97:0/399765487 >> v1:172.21.15.97:6789/0 conn(0x7f48bc0a98f0 legacy=0x7f48bc1d77d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:25.502 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.499+0000 7f48b8ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:26:25.503 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.499+0000 7f48b8ff9640 1 monclient: found mon.c 2024-03-25T18:26:25.503 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.499+0000 7f48b8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:25.503 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.499+0000 7f48b8ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.97:6790/0 2024-03-25T18:26:25.503 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.499+0000 7f48b8ff9640 1 -- 172.21.15.97:0/399765487 --> v1:172.21.15.97:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48bc0af960 con 0x7f48bc13f710 2024-03-25T18:26:25.503 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.499+0000 7f48b8ff9640 10 monclient: _finish_auth 0 2024-03-25T18:26:25.503 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.499+0000 7f48b8ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:26:25.503 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.499+0000 7f48b8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:25.504 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.499+0000 7f48c29e2640 5 monclient: authenticate success, global_id 4664 2024-03-25T18:26:25.504 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.499+0000 7f48c29e2640 10 monclient: _renew_subs 2024-03-25T18:26:25.504 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.499+0000 7f48c29e2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.97:6790/0 2024-03-25T18:26:25.504 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.499+0000 7f48c29e2640 1 -- 172.21.15.97:0/399765487 --> v1:172.21.15.97:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48bc0ae580 con 0x7f48bc13f710 2024-03-25T18:26:25.504 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.499+0000 7f48c29e2640 10 client.4664.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:26:25.504 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.499+0000 7f48c29e2640 10 monclient: _renew_subs 2024-03-25T18:26:25.504 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.499+0000 7f48c29e2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.97:6790/0 2024-03-25T18:26:25.504 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.499+0000 7f48c29e2640 1 -- 172.21.15.97:0/399765487 --> v1:172.21.15.97:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f48bc0aeb00 con 0x7f48bc13f710 2024-03-25T18:26:25.504 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.499+0000 7f48c29e2640 1 librados: init done 2024-03-25T18:26:25.504 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.500+0000 7f48c29e2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:25.504 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.500+0000 7f48c29e2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:25.504 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.500+0000 7f48c29e2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.97:6790/0 2024-03-25T18:26:25.504 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.500+0000 7f48c29e2640 1 -- 172.21.15.97:0/399765487 --> v1:172.21.15.97:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48bc0627b0 con 0x7f48bc13f710 2024-03-25T18:26:25.821 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.819+0000 7f48b8ff9640 1 -- 172.21.15.97:0/399765487 <== mon.2 v1:172.21.15.97:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f48b0004c70 con 0x7f48bc13f710 2024-03-25T18:26:25.821 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.819+0000 7f48b8ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:25.821 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.819+0000 7f48b8ff9640 1 -- 172.21.15.97:0/399765487 <== mon.2 v1:172.21.15.97:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f48b00050c0 con 0x7f48bc13f710 2024-03-25T18:26:25.822 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.819+0000 7f48b8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:25.822 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.819+0000 7f48b8ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:26:25.822 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.819+0000 7f48b8ff9640 10 monclient: dump: 2024-03-25T18:26:25.822 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:26:25.822 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:26:25.822 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:26:25.822 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:26:25.822 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:25.822 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:26:25.822 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:26:25.822 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:26:25.822 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:26:25.822 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:26:25.823 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.819+0000 7f48ba7fc640 4 set_mon_vals no callback set 2024-03-25T18:26:25.823 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.820+0000 7f48b8ff9640 1 -- 172.21.15.97:0/399765487 <== mon.2 v1:172.21.15.97:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2377622137 0 0) 0x7f48b00052e0 con 0x7f48bc13f710 2024-03-25T18:26:25.823 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.821+0000 7f48bbfff640 10 monclient: get_auth_request con 0x7f48a806de60 auth_method 0 2024-03-25T18:26:25.823 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.821+0000 7f48b8ff9640 10 client.4664.objecter ms_handle_connect 0x7f48a806de60 2024-03-25T18:26:25.828 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.827+0000 7f48b8ff9640 1 -- 172.21.15.97:0/399765487 <== mon.2 v1:172.21.15.97:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3078631805 0 0) 0x7f48b0054a50 con 0x7f48bc13f710 2024-03-25T18:26:25.829 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.827+0000 7f48b8ff9640 10 client.4664.objecter ms_dispatch 0x7f48bc000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:26:25.829 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.827+0000 7f48b8ff9640 3 client.4664.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:26:25.829 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.827+0000 7f48b8ff9640 3 client.4664.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:26:25.829 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.827+0000 7f48b8ff9640 20 client.4664.objecter dump_active .. 0 homeless 2024-03-25T18:26:25.829 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.827+0000 7f48b8ff9640 1 -- 172.21.15.97:0/399765487 <== mon.2 v1:172.21.15.97:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f48b0050560 con 0x7f48bc13f710 2024-03-25T18:26:25.829 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.827+0000 7f48b8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:25.829 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:25.827+0000 7f48b8ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:26:26.063 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:26.061+0000 7f48c29e2640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:26.089 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:26.061+0000 7f48c29e2640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:26.089 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:26.061+0000 7f48c29e2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.97:6790/0 2024-03-25T18:26:26.089 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:26.061+0000 7f48c29e2640 1 -- 172.21.15.97:0/399765487 --> v1:172.21.15.97:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f48bc0ad280 con 0x7f48bc13f710 2024-03-25T18:26:26.095 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:26.094+0000 7f48b8ff9640 1 -- 172.21.15.97:0/399765487 <== mon.2 v1:172.21.15.97:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 337131954) 0x7f48bc0ad280 con 0x7f48bc13f710 2024-03-25T18:26:26.095 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:26.094+0000 7f48b8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:26.096 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:26.094+0000 7f48b8ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:26:26.096 INFO:teuthology.misc.health.smithi097.stdout: 2024-03-25T18:26:26.096 INFO:teuthology.misc.health.smithi097.stdout:{"epoch":11,"fsid":"4c3d472c-79b8-486b-9e3d-2738d4306754","created":"2024-03-25T18:16:14.737309+0000","modified":"2024-03-25T18:26:18.750062+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:18.750062+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cb5152b9-262f-45c3-94b8-18d0e81df558","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"6daf8793-0838-473d-a7d6-9a0d2894c75d","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"136f1493-3b9c-4f43-a2e7-641482d69dae","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"397d538c-4f64-4daf-9e8b-662d09f91327","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4f1df08-663f-4bda-93f8-3878ab7c24b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c4b18d70-a3ce-4b15-9b70-eae5525bd761","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8dd4033a-de0e-4f8a-96dc-59f3c1f825b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0077722e-ffa6-4a3d-9891-929d9869b286","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:26:26.098 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:26.096+0000 7f48c29e2640 10 librados: watch_flush enter 2024-03-25T18:26:26.098 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:26.096+0000 7f48c29e2640 10 librados: watch_flush exit 2024-03-25T18:26:26.098 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:26.096+0000 7f48c29e2640 20 client.4664.objecter shutdown clearing up homeless session... 2024-03-25T18:26:26.099 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:26.096+0000 7f48c29e2640 10 client.4664.objecter successfully canceled tick 2024-03-25T18:26:26.099 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:26.096+0000 7f48c29e2640 1 -- 172.21.15.97:0/399765487 >> v1:172.21.15.98:6800/800387573 conn(0x7f48a806de60 legacy=0x7f48a8070320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:26.099 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:26.096+0000 7f48c29e2640 10 monclient: shutdown 2024-03-25T18:26:26.099 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:26.096+0000 7f48c29e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:26.099 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:26.096+0000 7f48c29e2640 1 -- 172.21.15.97:0/399765487 >> v1:172.21.15.97:6790/0 conn(0x7f48bc13f710 legacy=0x7f48bc0acb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:26.099 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:26.096+0000 7f48c29e2640 1 -- 172.21.15.97:0/399765487 shutdown_connections 2024-03-25T18:26:26.099 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:26.096+0000 7f48c29e2640 1 -- 172.21.15.97:0/399765487 >> 172.21.15.97:0/399765487 conn(0x7f48bc0b7ea0 msgr2=0x7f48bc074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:26.099 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:26.096+0000 7f48c29e2640 1 -- 172.21.15.97:0/399765487 shutdown_connections 2024-03-25T18:26:26.099 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:26.096+0000 7f48c29e2640 1 -- 172.21.15.97:0/399765487 wait complete. 2024-03-25T18:26:26.099 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:26.096+0000 7f48c29e2640 1 librados: shutdown 2024-03-25T18:26:26.109 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:26:32.916 DEBUG:teuthology.orchestra.run.smithi097:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:26:33.031 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.030+0000 7f5dbf081640 10 monclient: get_monmap_and_config 2024-03-25T18:26:33.031 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.030+0000 7f5dbf081640 10 monclient: build_initial_monmap 2024-03-25T18:26:33.032 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.030+0000 7f5dbf081640 10 monclient: monmap: 2024-03-25T18:26:33.032 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:26:33.032 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:26:33.032 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:26:33.031302+0000 2024-03-25T18:26:33.032 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:26:33.031302+0000 2024-03-25T18:26:33.032 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:33.032 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:26:33.032 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:26:33.032 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T18:26:33.032 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-b 2024-03-25T18:26:33.032 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:26:33.033 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.030+0000 7f5dbf081640 1 Processor -- start 2024-03-25T18:26:33.033 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.030+0000 7f5dbf081640 1 -- start start 2024-03-25T18:26:33.034 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.030+0000 7f5dbf081640 10 monclient: init 2024-03-25T18:26:33.034 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.031+0000 7f5dbf081640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:33.034 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.031+0000 7f5dbf081640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:26:33.034 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.031+0000 7f5dbf081640 10 monclient(hunting): picked mon.noname-a con 0x7f5db80b4d60 addr v1:172.21.15.97:6789/0 2024-03-25T18:26:33.034 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.031+0000 7f5dbf081640 10 monclient(hunting): picked mon.noname-b con 0x7f5db80b1730 addr v1:172.21.15.97:6790/0 2024-03-25T18:26:33.034 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.031+0000 7f5dbf081640 10 monclient(hunting): picked mon.noname-c con 0x7f5db814be90 addr v1:172.21.15.98:6789/0 2024-03-25T18:26:33.034 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.031+0000 7f5dbf081640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5db80af940 con 0x7f5db80b4d60 2024-03-25T18:26:33.034 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.031+0000 7f5dbf081640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5db80b0a70 con 0x7f5db814be90 2024-03-25T18:26:33.034 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.031+0000 7f5dbf081640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5db80b0d50 con 0x7f5db80b1730 2024-03-25T18:26:33.034 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.031+0000 7f5dbf081640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:33.034 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.031+0000 7f5dbf081640 10 monclient(hunting): authenticate will time out at 8205.307716s 2024-03-25T18:26:33.036 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.035+0000 7f5dbd61e640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f5db814be90 0x7f5db814e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.97:50808/0 (socket says 172.21.15.97:50808) 2024-03-25T18:26:33.036 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.035+0000 7f5dbd61e640 1 -- 172.21.15.97:0/284296005 learned_addr learned my addr 172.21.15.97:0/284296005 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:26:33.043 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.041+0000 7f5db77fe640 1 -- 172.21.15.97:0/284296005 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3891705487 0 0) 0x7f5db80b0a70 con 0x7f5db814be90 2024-03-25T18:26:33.043 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.042+0000 7f5db77fe640 10 monclient(hunting): _negotiate 2024-03-25T18:26:33.043 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.042+0000 7f5db77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:33.043 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.042+0000 7f5db77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:33.044 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.042+0000 7f5db77fe640 10 monclient(hunting): my global_id is 4690 2024-03-25T18:26:33.044 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.042+0000 7f5db77fe640 1 -- 172.21.15.97:0/284296005 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5da8003520 con 0x7f5db814be90 2024-03-25T18:26:33.050 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.046+0000 7f5db77fe640 1 -- 172.21.15.97:0/284296005 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1060770080 0 0) 0x7f5da8003520 con 0x7f5db814be90 2024-03-25T18:26:33.050 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.047+0000 7f5db77fe640 1 -- 172.21.15.97:0/284296005 >> v1:172.21.15.97:6790/0 conn(0x7f5db80b1730 legacy=0x7f5db80b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:26:33.050 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.047+0000 7f5db77fe640 1 -- 172.21.15.97:0/284296005 >> v1:172.21.15.97:6789/0 conn(0x7f5db80b4d60 legacy=0x7f5db813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:26:33.050 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.047+0000 7f5db77fe640 10 monclient: _finish_hunting 0 2024-03-25T18:26:33.050 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.047+0000 7f5db77fe640 1 monclient: found mon.noname-c 2024-03-25T18:26:33.050 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.047+0000 7f5db77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:33.050 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.047+0000 7f5db77fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.98:6789/0 2024-03-25T18:26:33.050 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.047+0000 7f5db77fe640 1 -- 172.21.15.97:0/284296005 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5db80b1030 con 0x7f5db814be90 2024-03-25T18:26:33.050 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.047+0000 7f5db77fe640 10 monclient: _finish_auth 0 2024-03-25T18:26:33.051 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.047+0000 7f5db77fe640 10 monclient: _check_auth_tickets 2024-03-25T18:26:33.051 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.047+0000 7f5db77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:33.051 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.047+0000 7f5db77fe640 1 -- 172.21.15.97:0/284296005 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f5db0001a10 con 0x7f5db814be90 2024-03-25T18:26:33.051 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.047+0000 7f5db77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:33.051 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.047+0000 7f5db77fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:26:33.051 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.047+0000 7f5db77fe640 10 monclient: dump: 2024-03-25T18:26:33.051 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:26:33.051 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:26:33.051 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:26:33.051 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:26:33.051 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:33.051 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:26:33.051 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:26:33.051 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:26:33.052 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:26:33.052 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:26:33.052 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.047+0000 7f5dbf081640 5 monclient: authenticate success, global_id 4690 2024-03-25T18:26:33.052 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.047+0000 7f5dbf081640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:26:33.058 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.056+0000 7f5db77fe640 1 -- 172.21.15.97:0/284296005 <== mon.1 v1:172.21.15.98:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5db0001be0 con 0x7f5db814be90 2024-03-25T18:26:33.058 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.056+0000 7f5db77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:33.058 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.056+0000 7f5db77fe640 1 -- 172.21.15.97:0/284296005 <== mon.1 v1:172.21.15.98:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f5db0002030 con 0x7f5db814be90 2024-03-25T18:26:33.058 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.056+0000 7f5db77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:33.058 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.056+0000 7f5db77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:26:33.058 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.056+0000 7f5db77fe640 10 monclient: dump: 2024-03-25T18:26:33.058 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:26:33.058 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:26:33.058 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:26:33.058 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:26:33.058 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:33.059 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:26:33.059 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:26:33.059 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:26:33.059 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:26:33.059 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:26:33.059 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.056+0000 7f5dbf081640 10 monclient: get_monmap_and_config success 2024-03-25T18:26:33.059 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.056+0000 7f5dbf081640 4 set_mon_vals no callback set 2024-03-25T18:26:33.061 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.057+0000 7f5dbf081640 10 monclient: shutdown 2024-03-25T18:26:33.061 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.057+0000 7f5dbf081640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:33.061 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.057+0000 7f5dbf081640 1 -- 172.21.15.97:0/284296005 >> v1:172.21.15.98:6789/0 conn(0x7f5db814be90 legacy=0x7f5db814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:33.061 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.057+0000 7f5dbf081640 1 -- 172.21.15.97:0/284296005 shutdown_connections 2024-03-25T18:26:33.061 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.057+0000 7f5dbf081640 1 -- 172.21.15.97:0/284296005 >> 172.21.15.97:0/284296005 conn(0x7f5db80617a0 msgr2=0x7f5db8061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:33.061 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.057+0000 7f5dbf081640 1 -- 172.21.15.97:0/284296005 shutdown_connections 2024-03-25T18:26:33.061 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.057+0000 7f5dbf081640 1 -- 172.21.15.97:0/284296005 wait complete. 2024-03-25T18:26:33.061 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.058+0000 7f5dbf081640 10 monclient: build_initial_monmap 2024-03-25T18:26:33.062 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.058+0000 7f5dbf081640 10 monclient: monmap: 2024-03-25T18:26:33.062 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:26:33.062 INFO:teuthology.misc.health.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:26:33.062 INFO:teuthology.misc.health.smithi097.stderr:last_changed 0.000000 2024-03-25T18:26:33.062 INFO:teuthology.misc.health.smithi097.stderr:created 0.000000 2024-03-25T18:26:33.062 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:33.062 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 1 2024-03-25T18:26:33.062 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:26:33.062 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T18:26:33.062 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-c 2024-03-25T18:26:33.062 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:26:33.062 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.058+0000 7f5dbf081640 1 librados: starting msgr at 2024-03-25T18:26:33.062 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.058+0000 7f5dbf081640 1 librados: starting objecter 2024-03-25T18:26:33.062 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.058+0000 7f5dbf081640 1 Processor -- start 2024-03-25T18:26:33.063 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.058+0000 7f5dbf081640 1 -- start start 2024-03-25T18:26:33.063 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.058+0000 7f5dbf081640 1 librados: setting wanted keys 2024-03-25T18:26:33.063 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.058+0000 7f5dbf081640 1 librados: calling monclient init 2024-03-25T18:26:33.063 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.058+0000 7f5dbf081640 10 monclient: init 2024-03-25T18:26:33.063 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.059+0000 7f5dbf081640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:33.063 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.059+0000 7f5dbf081640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:26:33.063 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.059+0000 7f5dbf081640 10 monclient(hunting): picked mon.noname-a con 0x7f5db80b1730 addr v1:172.21.15.97:6789/0 2024-03-25T18:26:33.063 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.059+0000 7f5dbf081640 10 monclient(hunting): picked mon.noname-b con 0x7f5db80b4d60 addr v1:172.21.15.98:6789/0 2024-03-25T18:26:33.063 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.059+0000 7f5dbf081640 10 monclient(hunting): picked mon.noname-c con 0x7f5db814be90 addr v1:172.21.15.97:6790/0 2024-03-25T18:26:33.063 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.059+0000 7f5dbf081640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5db813baa0 con 0x7f5db80b1730 2024-03-25T18:26:33.063 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.059+0000 7f5dbf081640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5db8139440 con 0x7f5db80b4d60 2024-03-25T18:26:33.063 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.059+0000 7f5dbf081640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5db8136700 con 0x7f5db814be90 2024-03-25T18:26:33.063 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.059+0000 7f5dbf081640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:33.063 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.059+0000 7f5dbf081640 10 monclient(hunting): authenticate will time out at 8205.335366s 2024-03-25T18:26:33.066 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.064+0000 7f5dbd61e640 1 --1- >> v1:172.21.15.97:6790/0 conn(0x7f5db814be90 0x7f5db81e2d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.97:6790/0 says I am v1:172.21.15.97:36172/0 (socket says 172.21.15.97:36172) 2024-03-25T18:26:33.066 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.064+0000 7f5dbd61e640 1 -- 172.21.15.97:0/3653257214 learned_addr learned my addr 172.21.15.97:0/3653257214 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:26:33.071 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.070+0000 7f5db57fa640 10 client.?.objecter ms_handle_connect 0x7f5db814be90 2024-03-25T18:26:33.071 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.070+0000 7f5db57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:33.074 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.072+0000 7f5db57fa640 10 client.?.objecter ms_handle_connect 0x7f5db80b1730 2024-03-25T18:26:33.074 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.072+0000 7f5db57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:33.075 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.073+0000 7f5db57fa640 10 client.?.objecter ms_handle_connect 0x7f5db80b4d60 2024-03-25T18:26:33.075 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.073+0000 7f5db57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:33.081 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.079+0000 7f5db57fa640 1 -- 172.21.15.97:0/3653257214 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3715501054 0 0) 0x7f5db8136700 con 0x7f5db814be90 2024-03-25T18:26:33.081 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.080+0000 7f5db57fa640 10 monclient(hunting): _negotiate 2024-03-25T18:26:33.081 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.080+0000 7f5db57fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:33.081 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.080+0000 7f5db57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:33.081 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.080+0000 7f5db57fa640 10 monclient(hunting): my global_id is 4667 2024-03-25T18:26:33.082 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.080+0000 7f5db57fa640 1 -- 172.21.15.97:0/3653257214 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d9c004eb0 con 0x7f5db814be90 2024-03-25T18:26:33.089 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.087+0000 7f5db57fa640 1 -- 172.21.15.97:0/3653257214 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1006292013 0 0) 0x7f5db8139440 con 0x7f5db80b4d60 2024-03-25T18:26:33.089 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.087+0000 7f5db57fa640 10 monclient(hunting): _negotiate 2024-03-25T18:26:33.089 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.087+0000 7f5db57fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:33.089 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.087+0000 7f5db57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:33.089 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.087+0000 7f5db57fa640 10 monclient(hunting): my global_id is 4693 2024-03-25T18:26:33.089 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.087+0000 7f5db57fa640 1 -- 172.21.15.97:0/3653257214 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5db8136700 con 0x7f5db80b4d60 2024-03-25T18:26:33.092 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.090+0000 7f5db57fa640 1 -- 172.21.15.97:0/3653257214 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4060067902 0 0) 0x7f5db813baa0 con 0x7f5db80b1730 2024-03-25T18:26:33.092 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.090+0000 7f5db57fa640 10 monclient(hunting): _negotiate 2024-03-25T18:26:33.092 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.090+0000 7f5db57fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:33.092 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.090+0000 7f5db57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:33.092 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.090+0000 7f5db57fa640 10 monclient(hunting): my global_id is 4683 2024-03-25T18:26:33.092 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.090+0000 7f5db57fa640 1 -- 172.21.15.97:0/3653257214 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5db8139440 con 0x7f5db80b1730 2024-03-25T18:26:33.111 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.109+0000 7f5db57fa640 1 -- 172.21.15.97:0/3653257214 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 780526901 0 0) 0x7f5db8136700 con 0x7f5db80b4d60 2024-03-25T18:26:33.111 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.109+0000 7f5db57fa640 1 -- 172.21.15.97:0/3653257214 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5db813baa0 con 0x7f5db80b4d60 2024-03-25T18:26:33.111 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.109+0000 7f5db57fa640 1 -- 172.21.15.97:0/3653257214 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f5da4001810 con 0x7f5db80b4d60 2024-03-25T18:26:33.111 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.109+0000 7f5db57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:33.111 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.109+0000 7f5db57fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:26:33.111 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.109+0000 7f5db57fa640 10 monclient(hunting): dump: 2024-03-25T18:26:33.111 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:26:33.112 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:26:33.112 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:26:33.112 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:26:33.112 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:33.112 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:26:33.112 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:26:33.112 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:26:33.112 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:26:33.112 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:26:33.112 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.109+0000 7f5db57fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:26:33.116 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.114+0000 7f5db57fa640 1 -- 172.21.15.97:0/3653257214 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3290212890 0 0) 0x7f5db8139440 con 0x7f5db80b1730 2024-03-25T18:26:33.116 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.114+0000 7f5db57fa640 1 -- 172.21.15.97:0/3653257214 --> v1:172.21.15.97:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5db8136700 con 0x7f5db80b1730 2024-03-25T18:26:33.116 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.114+0000 7f5db57fa640 1 -- 172.21.15.97:0/3653257214 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f5dac0017d0 con 0x7f5db80b1730 2024-03-25T18:26:33.116 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.114+0000 7f5db57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:33.116 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.114+0000 7f5db57fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:26:33.117 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.114+0000 7f5db57fa640 10 monclient(hunting): dump: 2024-03-25T18:26:33.117 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:26:33.117 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:26:33.117 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:26:33.117 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:26:33.117 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:33.118 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:26:33.118 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:26:33.118 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:26:33.118 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:26:33.118 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:26:33.122 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.120+0000 7f5db57fa640 1 -- 172.21.15.97:0/3653257214 <== mon.2 v1:172.21.15.97:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2756440319 0 0) 0x7f5d9c004eb0 con 0x7f5db814be90 2024-03-25T18:26:33.122 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.120+0000 7f5db57fa640 1 -- 172.21.15.97:0/3653257214 --> v1:172.21.15.97:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5db8139440 con 0x7f5db814be90 2024-03-25T18:26:33.122 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.120+0000 7f5db57fa640 1 -- 172.21.15.97:0/3653257214 <== mon.2 v1:172.21.15.97:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f5db0001f30 con 0x7f5db814be90 2024-03-25T18:26:33.122 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.120+0000 7f5db57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:33.122 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.120+0000 7f5db57fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:26:33.122 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.120+0000 7f5db57fa640 10 monclient(hunting): dump: 2024-03-25T18:26:33.122 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:26:33.122 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:26:33.122 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:26:33.122 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:26:33.123 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:33.123 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:26:33.123 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:26:33.123 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:26:33.123 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:26:33.123 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:26:33.127 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.125+0000 7f5db57fa640 1 -- 172.21.15.97:0/3653257214 <== mon.0 v1:172.21.15.97:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3453777180 0 0) 0x7f5db8136700 con 0x7f5db80b1730 2024-03-25T18:26:33.128 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.125+0000 7f5db57fa640 1 -- 172.21.15.97:0/3653257214 >> v1:172.21.15.97:6790/0 conn(0x7f5db814be90 legacy=0x7f5db81e2d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:33.128 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.125+0000 7f5db57fa640 1 -- 172.21.15.97:0/3653257214 >> v1:172.21.15.98:6789/0 conn(0x7f5db80b4d60 legacy=0x7f5db8132db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:33.128 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.125+0000 7f5db57fa640 10 monclient: _finish_hunting 0 2024-03-25T18:26:33.128 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.125+0000 7f5db57fa640 1 monclient: found mon.a 2024-03-25T18:26:33.128 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.125+0000 7f5db57fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:33.128 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.125+0000 7f5db57fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:26:33.128 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.125+0000 7f5db57fa640 1 -- 172.21.15.97:0/3653257214 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5db8136890 con 0x7f5db80b1730 2024-03-25T18:26:33.128 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.125+0000 7f5db57fa640 10 monclient: _finish_auth 0 2024-03-25T18:26:33.128 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.125+0000 7f5db57fa640 10 monclient: _check_auth_tickets 2024-03-25T18:26:33.128 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.125+0000 7f5db57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:33.128 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.125+0000 7f5dbf081640 5 monclient: authenticate success, global_id 4683 2024-03-25T18:26:33.129 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.125+0000 7f5dbf081640 10 monclient: _renew_subs 2024-03-25T18:26:33.129 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.125+0000 7f5dbf081640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:26:33.129 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.125+0000 7f5dbf081640 1 -- 172.21.15.97:0/3653257214 --> v1:172.21.15.97:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5db813bec0 con 0x7f5db80b1730 2024-03-25T18:26:33.129 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.125+0000 7f5dbf081640 10 client.4683.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:26:33.129 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.125+0000 7f5dbf081640 10 monclient: _renew_subs 2024-03-25T18:26:33.129 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.125+0000 7f5dbf081640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:26:33.129 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.125+0000 7f5dbf081640 1 -- 172.21.15.97:0/3653257214 --> v1:172.21.15.97:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5db81d8390 con 0x7f5db80b1730 2024-03-25T18:26:33.129 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.125+0000 7f5dbf081640 1 librados: init done 2024-03-25T18:26:33.130 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.126+0000 7f5dbf081640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:33.130 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.126+0000 7f5dbf081640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:33.130 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.126+0000 7f5dbf081640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:26:33.130 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.126+0000 7f5dbf081640 1 -- 172.21.15.97:0/3653257214 --> v1:172.21.15.97:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5db81401f0 con 0x7f5db80b1730 2024-03-25T18:26:33.429 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.427+0000 7f5db57fa640 1 -- 172.21.15.97:0/3653257214 <== mon.0 v1:172.21.15.97:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5dac001cc0 con 0x7f5db80b1730 2024-03-25T18:26:33.429 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.427+0000 7f5db57fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:33.429 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.427+0000 7f5db57fa640 1 -- 172.21.15.97:0/3653257214 <== mon.0 v1:172.21.15.97:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f5dac003790 con 0x7f5db80b1730 2024-03-25T18:26:33.430 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.427+0000 7f5db57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:33.430 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.427+0000 7f5db57fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:26:33.430 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.427+0000 7f5db57fa640 10 monclient: dump: 2024-03-25T18:26:33.430 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:26:33.430 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:26:33.430 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:26:33.430 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:26:33.430 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:33.430 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:26:33.430 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:26:33.430 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:26:33.430 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:26:33.430 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:26:33.431 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.427+0000 7f5db6ffd640 4 set_mon_vals no callback set 2024-03-25T18:26:33.431 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.428+0000 7f5db57fa640 1 -- 172.21.15.97:0/3653257214 <== mon.0 v1:172.21.15.97:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2377622137 0 0) 0x7f5dac019760 con 0x7f5db80b1730 2024-03-25T18:26:33.431 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.429+0000 7f5db7fff640 10 monclient: get_auth_request con 0x7f5d9c06e110 auth_method 0 2024-03-25T18:26:33.431 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.429+0000 7f5db57fa640 10 client.4683.objecter ms_handle_connect 0x7f5d9c06e110 2024-03-25T18:26:33.436 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.434+0000 7f5db57fa640 1 -- 172.21.15.97:0/3653257214 <== mon.0 v1:172.21.15.97:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3078631805 0 0) 0x7f5dac0534b0 con 0x7f5db80b1730 2024-03-25T18:26:33.436 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.434+0000 7f5db57fa640 10 client.4683.objecter ms_dispatch 0x7f5db8000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:26:33.437 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.434+0000 7f5db57fa640 3 client.4683.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:26:33.437 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.434+0000 7f5db57fa640 3 client.4683.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:26:33.437 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.434+0000 7f5db57fa640 20 client.4683.objecter dump_active .. 0 homeless 2024-03-25T18:26:33.437 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.434+0000 7f5db57fa640 1 -- 172.21.15.97:0/3653257214 <== mon.0 v1:172.21.15.97:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5dac04efc0 con 0x7f5db80b1730 2024-03-25T18:26:33.437 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.434+0000 7f5db57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:33.437 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.434+0000 7f5db57fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:26:33.662 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.660+0000 7f5dbf081640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:33.662 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.660+0000 7f5dbf081640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:33.663 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.660+0000 7f5dbf081640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:26:33.663 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.660+0000 7f5dbf081640 1 -- 172.21.15.97:0/3653257214 --> v1:172.21.15.97:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5db80b22c0 con 0x7f5db80b1730 2024-03-25T18:26:33.685 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.683+0000 7f5db57fa640 1 -- 172.21.15.97:0/3653257214 <== mon.0 v1:172.21.15.97:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 337131954) 0x7f5dac050ef0 con 0x7f5db80b1730 2024-03-25T18:26:33.685 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.683+0000 7f5db57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:33.685 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.683+0000 7f5db57fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:26:33.685 INFO:teuthology.misc.health.smithi097.stdout: 2024-03-25T18:26:33.686 INFO:teuthology.misc.health.smithi097.stdout:{"epoch":11,"fsid":"4c3d472c-79b8-486b-9e3d-2738d4306754","created":"2024-03-25T18:16:14.737309+0000","modified":"2024-03-25T18:26:18.750062+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:18.750062+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cb5152b9-262f-45c3-94b8-18d0e81df558","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"6daf8793-0838-473d-a7d6-9a0d2894c75d","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"136f1493-3b9c-4f43-a2e7-641482d69dae","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"397d538c-4f64-4daf-9e8b-662d09f91327","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4f1df08-663f-4bda-93f8-3878ab7c24b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c4b18d70-a3ce-4b15-9b70-eae5525bd761","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8dd4033a-de0e-4f8a-96dc-59f3c1f825b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0077722e-ffa6-4a3d-9891-929d9869b286","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:26:33.687 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.685+0000 7f5dbf081640 10 librados: watch_flush enter 2024-03-25T18:26:33.687 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.685+0000 7f5dbf081640 10 librados: watch_flush exit 2024-03-25T18:26:33.688 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.685+0000 7f5dbf081640 20 client.4683.objecter shutdown clearing up homeless session... 2024-03-25T18:26:33.688 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.685+0000 7f5dbf081640 10 client.4683.objecter successfully canceled tick 2024-03-25T18:26:33.688 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.685+0000 7f5dbf081640 1 -- 172.21.15.97:0/3653257214 >> v1:172.21.15.98:6800/800387573 conn(0x7f5d9c06e110 legacy=0x7f5d9c0705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:33.688 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.685+0000 7f5dbf081640 10 monclient: shutdown 2024-03-25T18:26:33.688 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.685+0000 7f5dbf081640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:33.688 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.685+0000 7f5dbf081640 1 -- 172.21.15.97:0/3653257214 >> v1:172.21.15.97:6789/0 conn(0x7f5db80b1730 legacy=0x7f5db81d7aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:33.688 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.685+0000 7f5dbf081640 1 -- 172.21.15.97:0/3653257214 shutdown_connections 2024-03-25T18:26:33.688 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.685+0000 7f5dbf081640 1 -- 172.21.15.97:0/3653257214 >> 172.21.15.97:0/3653257214 conn(0x7f5db80617a0 msgr2=0x7f5db80b7c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:33.688 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.686+0000 7f5dbf081640 1 -- 172.21.15.97:0/3653257214 shutdown_connections 2024-03-25T18:26:33.689 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.686+0000 7f5dbf081640 1 -- 172.21.15.97:0/3653257214 wait complete. 2024-03-25T18:26:33.689 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:33.686+0000 7f5dbf081640 1 librados: shutdown 2024-03-25T18:26:33.698 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:26:38.928 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:10:40.967 34931== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:38.938 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:10:40.976 34931== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:38.976 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:10:41.015 34931== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:38.993 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:10:41.031 34935== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:38.998 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:10:41.036 34935== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:38.998 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:10:41.041 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:39.005 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:10:41.048 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:39.026 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:10:41.069 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:39.031 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:10:41.069 34935== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:40.506 DEBUG:teuthology.orchestra.run.smithi097:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:26:40.620 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.618+0000 7efe36406640 10 monclient: get_monmap_and_config 2024-03-25T18:26:40.621 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.618+0000 7efe36406640 10 monclient: build_initial_monmap 2024-03-25T18:26:40.621 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.619+0000 7efe36406640 10 monclient: monmap: 2024-03-25T18:26:40.621 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:26:40.621 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:26:40.621 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:26:40.620250+0000 2024-03-25T18:26:40.621 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:26:40.620250+0000 2024-03-25T18:26:40.621 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:40.621 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:26:40.621 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:26:40.621 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T18:26:40.621 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-b 2024-03-25T18:26:40.621 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:26:40.622 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.619+0000 7efe36406640 1 Processor -- start 2024-03-25T18:26:40.623 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.619+0000 7efe36406640 1 -- start start 2024-03-25T18:26:40.623 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.619+0000 7efe36406640 10 monclient: init 2024-03-25T18:26:40.623 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.620+0000 7efe36406640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:40.623 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.620+0000 7efe36406640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:26:40.623 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.620+0000 7efe36406640 10 monclient(hunting): picked mon.noname-c con 0x7efe300a98f0 addr v1:172.21.15.98:6789/0 2024-03-25T18:26:40.623 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.620+0000 7efe36406640 10 monclient(hunting): picked mon.noname-a con 0x7efe3013f710 addr v1:172.21.15.97:6789/0 2024-03-25T18:26:40.623 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.620+0000 7efe36406640 10 monclient(hunting): picked mon.noname-b con 0x7efe301430c0 addr v1:172.21.15.97:6790/0 2024-03-25T18:26:40.623 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.620+0000 7efe36406640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe30142520 con 0x7efe3013f710 2024-03-25T18:26:40.623 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.620+0000 7efe36406640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe300b28a0 con 0x7efe300a98f0 2024-03-25T18:26:40.623 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.620+0000 7efe36406640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe300b2a30 con 0x7efe301430c0 2024-03-25T18:26:40.623 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.620+0000 7efe36406640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:40.624 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.620+0000 7efe36406640 10 monclient(hunting): authenticate will time out at 8212.896567s 2024-03-25T18:26:40.631 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.629+0000 7efe349a3640 1 --1- >> v1:172.21.15.97:6790/0 conn(0x7efe301430c0 0x7efe3014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6790/0 says I am v1:172.21.15.97:33796/0 (socket says 172.21.15.97:33796) 2024-03-25T18:26:40.631 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.629+0000 7efe349a3640 1 -- 172.21.15.97:0/3240521800 learned_addr learned my addr 172.21.15.97:0/3240521800 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:26:40.640 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.639+0000 7efe2effd640 1 -- 172.21.15.97:0/3240521800 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2508035602 0 0) 0x7efe300b2a30 con 0x7efe301430c0 2024-03-25T18:26:40.641 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.639+0000 7efe2effd640 10 monclient(hunting): _negotiate 2024-03-25T18:26:40.641 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.639+0000 7efe2effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:40.641 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.639+0000 7efe2effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:40.641 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.639+0000 7efe2effd640 10 monclient(hunting): my global_id is 4670 2024-03-25T18:26:40.641 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.639+0000 7efe2effd640 1 -- 172.21.15.97:0/3240521800 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe18003520 con 0x7efe301430c0 2024-03-25T18:26:40.644 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.642+0000 7efe2effd640 1 -- 172.21.15.97:0/3240521800 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1496113056 0 0) 0x7efe300b28a0 con 0x7efe300a98f0 2024-03-25T18:26:40.644 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.642+0000 7efe2effd640 10 monclient(hunting): _negotiate 2024-03-25T18:26:40.644 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.642+0000 7efe2effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:40.644 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.642+0000 7efe2effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:40.644 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.642+0000 7efe2effd640 10 monclient(hunting): my global_id is 4696 2024-03-25T18:26:40.645 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.642+0000 7efe2effd640 1 -- 172.21.15.97:0/3240521800 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe300b2a30 con 0x7efe300a98f0 2024-03-25T18:26:40.654 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.652+0000 7efe2effd640 1 -- 172.21.15.97:0/3240521800 <== mon.2 v1:172.21.15.97:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 284526932 0 0) 0x7efe18003520 con 0x7efe301430c0 2024-03-25T18:26:40.654 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.652+0000 7efe2effd640 1 -- 172.21.15.97:0/3240521800 >> v1:172.21.15.98:6789/0 conn(0x7efe300a98f0 legacy=0x7efe300a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:40.654 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.652+0000 7efe2effd640 1 -- 172.21.15.97:0/3240521800 >> v1:172.21.15.97:6789/0 conn(0x7efe3013f710 legacy=0x7efe301429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:40.654 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.652+0000 7efe2effd640 10 monclient: _finish_hunting 0 2024-03-25T18:26:40.655 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.652+0000 7efe2effd640 1 monclient: found mon.noname-b 2024-03-25T18:26:40.655 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.652+0000 7efe2effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:40.655 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.652+0000 7efe2effd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.97:6790/0 2024-03-25T18:26:40.655 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.652+0000 7efe2effd640 1 -- 172.21.15.97:0/3240521800 --> v1:172.21.15.97:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe300b44c0 con 0x7efe301430c0 2024-03-25T18:26:40.655 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.652+0000 7efe2effd640 10 monclient: _finish_auth 0 2024-03-25T18:26:40.655 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.652+0000 7efe2effd640 10 monclient: _check_auth_tickets 2024-03-25T18:26:40.655 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.652+0000 7efe2effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:40.655 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.652+0000 7efe2effd640 1 -- 172.21.15.97:0/3240521800 <== mon.2 v1:172.21.15.97:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7efe28001a10 con 0x7efe301430c0 2024-03-25T18:26:40.655 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.652+0000 7efe2effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:40.655 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.652+0000 7efe2effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:26:40.655 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.652+0000 7efe2effd640 10 monclient: dump: 2024-03-25T18:26:40.655 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:26:40.655 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:26:40.656 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:26:40.656 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:26:40.656 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:40.656 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:26:40.656 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:26:40.656 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:26:40.656 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:26:40.656 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:26:40.656 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.652+0000 7efe36406640 5 monclient: authenticate success, global_id 4670 2024-03-25T18:26:40.656 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.652+0000 7efe36406640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:26:40.675 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.673+0000 7efe2effd640 1 -- 172.21.15.97:0/3240521800 <== mon.2 v1:172.21.15.97:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efe28001b70 con 0x7efe301430c0 2024-03-25T18:26:40.675 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.673+0000 7efe2effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:40.675 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.673+0000 7efe2effd640 1 -- 172.21.15.97:0/3240521800 <== mon.2 v1:172.21.15.97:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7efe28001f80 con 0x7efe301430c0 2024-03-25T18:26:40.675 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.673+0000 7efe36406640 10 monclient: get_monmap_and_config success 2024-03-25T18:26:40.676 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.673+0000 7efe36406640 4 set_mon_vals no callback set 2024-03-25T18:26:40.676 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.673+0000 7efe2effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:40.676 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.673+0000 7efe2effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:26:40.676 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.673+0000 7efe2effd640 10 monclient: dump: 2024-03-25T18:26:40.676 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:26:40.676 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:26:40.676 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:26:40.676 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:26:40.676 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:40.676 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:26:40.676 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:26:40.676 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:26:40.676 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:26:40.677 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:26:40.679 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.674+0000 7efe36406640 10 monclient: shutdown 2024-03-25T18:26:40.679 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.674+0000 7efe36406640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:40.679 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.674+0000 7efe36406640 1 -- 172.21.15.97:0/3240521800 >> v1:172.21.15.97:6790/0 conn(0x7efe301430c0 legacy=0x7efe3014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:40.679 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.674+0000 7efe36406640 1 -- 172.21.15.97:0/3240521800 shutdown_connections 2024-03-25T18:26:40.679 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.674+0000 7efe36406640 1 -- 172.21.15.97:0/3240521800 >> 172.21.15.97:0/3240521800 conn(0x7efe300b7ea0 msgr2=0x7efe300b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:40.679 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.674+0000 7efe36406640 1 -- 172.21.15.97:0/3240521800 shutdown_connections 2024-03-25T18:26:40.680 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.674+0000 7efe36406640 1 -- 172.21.15.97:0/3240521800 wait complete. 2024-03-25T18:26:40.680 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.675+0000 7efe36406640 10 monclient: build_initial_monmap 2024-03-25T18:26:40.680 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.675+0000 7efe36406640 10 monclient: monmap: 2024-03-25T18:26:40.680 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:26:40.680 INFO:teuthology.misc.health.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:26:40.680 INFO:teuthology.misc.health.smithi097.stderr:last_changed 0.000000 2024-03-25T18:26:40.680 INFO:teuthology.misc.health.smithi097.stderr:created 0.000000 2024-03-25T18:26:40.680 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:40.680 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 1 2024-03-25T18:26:40.680 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:26:40.680 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T18:26:40.680 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-c 2024-03-25T18:26:40.680 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:26:40.681 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.675+0000 7efe36406640 1 librados: starting msgr at 2024-03-25T18:26:40.681 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.675+0000 7efe36406640 1 librados: starting objecter 2024-03-25T18:26:40.681 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.676+0000 7efe36406640 1 Processor -- start 2024-03-25T18:26:40.681 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.676+0000 7efe36406640 1 -- start start 2024-03-25T18:26:40.681 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.676+0000 7efe36406640 1 librados: setting wanted keys 2024-03-25T18:26:40.681 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.676+0000 7efe36406640 1 librados: calling monclient init 2024-03-25T18:26:40.681 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.676+0000 7efe36406640 10 monclient: init 2024-03-25T18:26:40.681 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.676+0000 7efe36406640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:40.681 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.676+0000 7efe36406640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:26:40.681 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.676+0000 7efe36406640 10 monclient(hunting): picked mon.noname-a con 0x7efe300a98f0 addr v1:172.21.15.97:6789/0 2024-03-25T18:26:40.681 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.676+0000 7efe36406640 10 monclient(hunting): picked mon.noname-b con 0x7efe3013f710 addr v1:172.21.15.98:6789/0 2024-03-25T18:26:40.681 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.677+0000 7efe36406640 10 monclient(hunting): picked mon.noname-c con 0x7efe301430c0 addr v1:172.21.15.97:6790/0 2024-03-25T18:26:40.681 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.677+0000 7efe36406640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe300b6f90 con 0x7efe300a98f0 2024-03-25T18:26:40.682 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.677+0000 7efe36406640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe3013b440 con 0x7efe3013f710 2024-03-25T18:26:40.682 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.677+0000 7efe36406640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe300b6b40 con 0x7efe301430c0 2024-03-25T18:26:40.682 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.677+0000 7efe36406640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:40.682 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.677+0000 7efe36406640 10 monclient(hunting): authenticate will time out at 8212.952997s 2024-03-25T18:26:40.684 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.682+0000 7efe2f7fe640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7efe3013f710 0x7efe301db890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.97:56564/0 (socket says 172.21.15.97:56564) 2024-03-25T18:26:40.684 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.682+0000 7efe2f7fe640 1 -- 172.21.15.97:0/3225977987 learned_addr learned my addr 172.21.15.97:0/3225977987 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:26:40.685 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.684+0000 7efe2cff9640 10 client.?.objecter ms_handle_connect 0x7efe3013f710 2024-03-25T18:26:40.686 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.684+0000 7efe2cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:40.697 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.695+0000 7efe2cff9640 10 client.?.objecter ms_handle_connect 0x7efe300a98f0 2024-03-25T18:26:40.697 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.695+0000 7efe2cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:40.698 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.696+0000 7efe2cff9640 1 -- 172.21.15.97:0/3225977987 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4106123801 0 0) 0x7efe3013b440 con 0x7efe3013f710 2024-03-25T18:26:40.698 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.696+0000 7efe2cff9640 10 monclient(hunting): _negotiate 2024-03-25T18:26:40.698 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.696+0000 7efe2cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:40.698 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.696+0000 7efe2cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:40.698 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.696+0000 7efe2cff9640 10 monclient(hunting): my global_id is 4699 2024-03-25T18:26:40.698 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.696+0000 7efe2cff9640 1 -- 172.21.15.97:0/3225977987 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe1c004eb0 con 0x7efe3013f710 2024-03-25T18:26:40.701 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.699+0000 7efe2cff9640 10 client.?.objecter ms_handle_connect 0x7efe301430c0 2024-03-25T18:26:40.701 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.699+0000 7efe2cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:40.702 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.701+0000 7efe2cff9640 1 -- 172.21.15.97:0/3225977987 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3897333816 0 0) 0x7efe300b6f90 con 0x7efe300a98f0 2024-03-25T18:26:40.702 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.701+0000 7efe2cff9640 10 monclient(hunting): _negotiate 2024-03-25T18:26:40.702 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.701+0000 7efe2cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:40.702 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.701+0000 7efe2cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:40.702 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.701+0000 7efe2cff9640 10 monclient(hunting): my global_id is 4689 2024-03-25T18:26:40.702 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.701+0000 7efe2cff9640 1 -- 172.21.15.97:0/3225977987 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe3013b440 con 0x7efe300a98f0 2024-03-25T18:26:40.709 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.707+0000 7efe2cff9640 1 -- 172.21.15.97:0/3225977987 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3230653610 0 0) 0x7efe300b6b40 con 0x7efe301430c0 2024-03-25T18:26:40.709 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.707+0000 7efe2cff9640 10 monclient(hunting): _negotiate 2024-03-25T18:26:40.709 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.707+0000 7efe2cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:40.709 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.707+0000 7efe2cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:40.709 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.707+0000 7efe2cff9640 10 monclient(hunting): my global_id is 4673 2024-03-25T18:26:40.709 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.707+0000 7efe2cff9640 1 -- 172.21.15.97:0/3225977987 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe300b6f90 con 0x7efe301430c0 2024-03-25T18:26:40.711 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.709+0000 7efe2cff9640 1 -- 172.21.15.97:0/3225977987 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 490870673 0 0) 0x7efe1c004eb0 con 0x7efe3013f710 2024-03-25T18:26:40.711 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.709+0000 7efe2cff9640 1 -- 172.21.15.97:0/3225977987 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efe300b6b40 con 0x7efe3013f710 2024-03-25T18:26:40.711 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.709+0000 7efe2cff9640 1 -- 172.21.15.97:0/3225977987 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7efe24001a30 con 0x7efe3013f710 2024-03-25T18:26:40.711 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.709+0000 7efe2cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:40.711 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.709+0000 7efe2cff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:26:40.711 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.709+0000 7efe2cff9640 10 monclient(hunting): dump: 2024-03-25T18:26:40.711 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:26:40.712 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:26:40.712 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:26:40.712 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:26:40.712 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:40.712 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:26:40.712 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:26:40.712 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:26:40.712 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:26:40.712 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:26:40.712 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.709+0000 7efe2cff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:26:40.717 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.715+0000 7efe2cff9640 1 -- 172.21.15.97:0/3225977987 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3364546164 0 0) 0x7efe3013b440 con 0x7efe300a98f0 2024-03-25T18:26:40.717 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.715+0000 7efe2cff9640 1 -- 172.21.15.97:0/3225977987 --> v1:172.21.15.97:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efe1c004eb0 con 0x7efe300a98f0 2024-03-25T18:26:40.717 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.715+0000 7efe2cff9640 1 -- 172.21.15.97:0/3225977987 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7efe20001930 con 0x7efe300a98f0 2024-03-25T18:26:40.717 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.715+0000 7efe2cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:40.717 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.715+0000 7efe2cff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:26:40.717 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.715+0000 7efe2cff9640 10 monclient(hunting): dump: 2024-03-25T18:26:40.717 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:26:40.717 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:26:40.717 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:26:40.717 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:26:40.717 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:40.717 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:26:40.718 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:26:40.718 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:26:40.718 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:26:40.718 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:26:40.722 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.721+0000 7efe2cff9640 1 -- 172.21.15.97:0/3225977987 <== mon.2 v1:172.21.15.97:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2815479430 0 0) 0x7efe300b6f90 con 0x7efe301430c0 2024-03-25T18:26:40.722 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.721+0000 7efe2cff9640 1 -- 172.21.15.97:0/3225977987 --> v1:172.21.15.97:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efe3013b440 con 0x7efe301430c0 2024-03-25T18:26:40.723 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.721+0000 7efe2cff9640 1 -- 172.21.15.97:0/3225977987 <== mon.2 v1:172.21.15.97:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7efe28001e80 con 0x7efe301430c0 2024-03-25T18:26:40.723 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.721+0000 7efe2cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:40.723 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.721+0000 7efe2cff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:26:40.723 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.721+0000 7efe2cff9640 10 monclient(hunting): dump: 2024-03-25T18:26:40.723 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:26:40.723 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:26:40.723 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:26:40.723 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:26:40.723 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:40.723 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:26:40.723 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:26:40.723 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:26:40.723 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:26:40.723 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:26:40.725 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.721+0000 7efe2cff9640 1 -- 172.21.15.97:0/3225977987 <== mon.0 v1:172.21.15.97:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1101046624 0 0) 0x7efe1c004eb0 con 0x7efe300a98f0 2024-03-25T18:26:40.725 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.721+0000 7efe2cff9640 1 -- 172.21.15.97:0/3225977987 >> v1:172.21.15.97:6790/0 conn(0x7efe301430c0 legacy=0x7efe301dbfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:40.725 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.721+0000 7efe2cff9640 1 -- 172.21.15.97:0/3225977987 >> v1:172.21.15.98:6789/0 conn(0x7efe3013f710 legacy=0x7efe301db890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:40.725 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.721+0000 7efe2cff9640 10 monclient: _finish_hunting 0 2024-03-25T18:26:40.725 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.721+0000 7efe2cff9640 1 monclient: found mon.a 2024-03-25T18:26:40.725 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.721+0000 7efe2cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:40.725 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.721+0000 7efe2cff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:26:40.725 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.721+0000 7efe2cff9640 1 -- 172.21.15.97:0/3225977987 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe300b6cd0 con 0x7efe300a98f0 2024-03-25T18:26:40.725 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.721+0000 7efe2cff9640 10 monclient: _finish_auth 0 2024-03-25T18:26:40.725 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.721+0000 7efe2cff9640 10 monclient: _check_auth_tickets 2024-03-25T18:26:40.725 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.721+0000 7efe2cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:40.726 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.721+0000 7efe36406640 5 monclient: authenticate success, global_id 4689 2024-03-25T18:26:40.726 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.722+0000 7efe36406640 10 monclient: _renew_subs 2024-03-25T18:26:40.726 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.722+0000 7efe36406640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:26:40.726 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.722+0000 7efe36406640 1 -- 172.21.15.97:0/3225977987 --> v1:172.21.15.97:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe301356a0 con 0x7efe300a98f0 2024-03-25T18:26:40.726 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.722+0000 7efe36406640 10 client.4689.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:26:40.726 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.722+0000 7efe36406640 10 monclient: _renew_subs 2024-03-25T18:26:40.726 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.722+0000 7efe36406640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:26:40.726 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.722+0000 7efe36406640 1 -- 172.21.15.97:0/3225977987 --> v1:172.21.15.97:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efe3014c270 con 0x7efe300a98f0 2024-03-25T18:26:40.726 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.722+0000 7efe36406640 1 librados: init done 2024-03-25T18:26:40.726 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.722+0000 7efe36406640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:40.726 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.722+0000 7efe36406640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:40.726 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.722+0000 7efe36406640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:26:40.726 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:40.722+0000 7efe36406640 1 -- 172.21.15.97:0/3225977987 --> v1:172.21.15.97:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe300627b0 con 0x7efe300a98f0 2024-03-25T18:26:41.030 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:41.028+0000 7efe2cff9640 1 -- 172.21.15.97:0/3225977987 <== mon.0 v1:172.21.15.97:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efe20003610 con 0x7efe300a98f0 2024-03-25T18:26:41.030 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:41.028+0000 7efe2cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:41.031 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:41.028+0000 7efe2cff9640 1 -- 172.21.15.97:0/3225977987 <== mon.0 v1:172.21.15.97:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7efe20003920 con 0x7efe300a98f0 2024-03-25T18:26:41.031 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:41.028+0000 7efe2cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:41.031 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:41.028+0000 7efe2cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:26:41.031 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:41.028+0000 7efe2cff9640 10 monclient: dump: 2024-03-25T18:26:41.031 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:26:41.031 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:26:41.031 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:26:41.031 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:26:41.031 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:41.031 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:26:41.031 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:26:41.031 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:26:41.031 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:26:41.031 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:26:41.032 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:41.028+0000 7efe2dffb640 4 set_mon_vals no callback set 2024-03-25T18:26:41.032 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:41.029+0000 7efe2cff9640 1 -- 172.21.15.97:0/3225977987 <== mon.0 v1:172.21.15.97:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2377622137 0 0) 0x7efe200198f0 con 0x7efe300a98f0 2024-03-25T18:26:41.032 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:41.030+0000 7efe2f7fe640 10 monclient: get_auth_request con 0x7efe1c06e160 auth_method 0 2024-03-25T18:26:41.032 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:41.030+0000 7efe2cff9640 10 client.4689.objecter ms_handle_connect 0x7efe1c06e160 2024-03-25T18:26:41.037 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:41.035+0000 7efe2cff9640 1 -- 172.21.15.97:0/3225977987 <== mon.0 v1:172.21.15.97:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3078631805 0 0) 0x7efe20053590 con 0x7efe300a98f0 2024-03-25T18:26:41.037 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:41.035+0000 7efe2cff9640 10 client.4689.objecter ms_dispatch 0x7efe30000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:26:41.037 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:41.035+0000 7efe2cff9640 3 client.4689.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:26:41.037 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:41.035+0000 7efe2cff9640 3 client.4689.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:26:41.037 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:41.035+0000 7efe2cff9640 20 client.4689.objecter dump_active .. 0 homeless 2024-03-25T18:26:41.037 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:41.035+0000 7efe2cff9640 1 -- 172.21.15.97:0/3225977987 <== mon.0 v1:172.21.15.97:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7efe2004eed0 con 0x7efe300a98f0 2024-03-25T18:26:41.037 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:41.035+0000 7efe2cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:41.037 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:41.035+0000 7efe2cff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:26:41.255 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:41.254+0000 7efe36406640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:41.256 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:41.254+0000 7efe36406640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:41.256 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:41.254+0000 7efe36406640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:26:41.256 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:41.254+0000 7efe36406640 1 -- 172.21.15.97:0/3225977987 --> v1:172.21.15.97:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efe30069f40 con 0x7efe300a98f0 2024-03-25T18:26:41.308 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:41.306+0000 7efe2cff9640 1 -- 172.21.15.97:0/3225977987 <== mon.0 v1:172.21.15.97:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 337131954) 0x7efe20050fd0 con 0x7efe300a98f0 2024-03-25T18:26:41.308 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:41.306+0000 7efe2cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:41.308 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:41.306+0000 7efe2cff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:26:41.308 INFO:teuthology.misc.health.smithi097.stdout: 2024-03-25T18:26:41.308 INFO:teuthology.misc.health.smithi097.stdout:{"epoch":11,"fsid":"4c3d472c-79b8-486b-9e3d-2738d4306754","created":"2024-03-25T18:16:14.737309+0000","modified":"2024-03-25T18:26:18.750062+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:18.750062+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cb5152b9-262f-45c3-94b8-18d0e81df558","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"6daf8793-0838-473d-a7d6-9a0d2894c75d","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"136f1493-3b9c-4f43-a2e7-641482d69dae","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"397d538c-4f64-4daf-9e8b-662d09f91327","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4f1df08-663f-4bda-93f8-3878ab7c24b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c4b18d70-a3ce-4b15-9b70-eae5525bd761","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8dd4033a-de0e-4f8a-96dc-59f3c1f825b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0077722e-ffa6-4a3d-9891-929d9869b286","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:26:41.311 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:41.308+0000 7efe36406640 10 librados: watch_flush enter 2024-03-25T18:26:41.311 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:41.308+0000 7efe36406640 10 librados: watch_flush exit 2024-03-25T18:26:41.311 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:41.308+0000 7efe36406640 20 client.4689.objecter shutdown clearing up homeless session... 2024-03-25T18:26:41.311 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:41.308+0000 7efe36406640 10 client.4689.objecter successfully canceled tick 2024-03-25T18:26:41.311 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:41.308+0000 7efe36406640 1 -- 172.21.15.97:0/3225977987 >> v1:172.21.15.98:6800/800387573 conn(0x7efe1c06e160 legacy=0x7efe1c070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:41.311 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:41.308+0000 7efe36406640 10 monclient: shutdown 2024-03-25T18:26:41.311 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:41.308+0000 7efe36406640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:41.311 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:41.308+0000 7efe36406640 1 -- 172.21.15.97:0/3225977987 >> v1:172.21.15.97:6789/0 conn(0x7efe300a98f0 legacy=0x7efe30134f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:41.311 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:41.308+0000 7efe36406640 1 -- 172.21.15.97:0/3225977987 shutdown_connections 2024-03-25T18:26:41.311 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:41.308+0000 7efe36406640 1 -- 172.21.15.97:0/3225977987 >> 172.21.15.97:0/3225977987 conn(0x7efe300b7ea0 msgr2=0x7efe30074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:41.311 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:41.309+0000 7efe36406640 1 -- 172.21.15.97:0/3225977987 shutdown_connections 2024-03-25T18:26:41.311 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:41.309+0000 7efe36406640 1 -- 172.21.15.97:0/3225977987 wait complete. 2024-03-25T18:26:41.311 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:41.309+0000 7efe36406640 1 librados: shutdown 2024-03-25T18:26:41.323 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:26:48.131 DEBUG:teuthology.orchestra.run.smithi097:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:26:48.247 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.245+0000 7f3e46a16640 10 monclient: get_monmap_and_config 2024-03-25T18:26:48.247 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.245+0000 7f3e46a16640 10 monclient: build_initial_monmap 2024-03-25T18:26:48.247 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.245+0000 7f3e46a16640 10 monclient: monmap: 2024-03-25T18:26:48.247 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:26:48.247 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:26:48.247 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:26:48.246868+0000 2024-03-25T18:26:48.248 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:26:48.246868+0000 2024-03-25T18:26:48.248 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:48.248 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:26:48.248 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:26:48.248 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T18:26:48.248 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-b 2024-03-25T18:26:48.248 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:26:48.249 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.246+0000 7f3e46a16640 1 Processor -- start 2024-03-25T18:26:48.249 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.246+0000 7f3e46a16640 1 -- start start 2024-03-25T18:26:48.249 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.246+0000 7f3e46a16640 10 monclient: init 2024-03-25T18:26:48.249 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.247+0000 7f3e46a16640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:48.249 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.247+0000 7f3e46a16640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:26:48.250 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.247+0000 7f3e46a16640 10 monclient(hunting): picked mon.noname-a con 0x7f3e400a98f0 addr v1:172.21.15.97:6789/0 2024-03-25T18:26:48.250 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.247+0000 7f3e46a16640 10 monclient(hunting): picked mon.noname-b con 0x7f3e4013f710 addr v1:172.21.15.97:6790/0 2024-03-25T18:26:48.250 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.247+0000 7f3e46a16640 10 monclient(hunting): picked mon.noname-c con 0x7f3e401430c0 addr v1:172.21.15.98:6789/0 2024-03-25T18:26:48.250 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.247+0000 7f3e46a16640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e40142520 con 0x7f3e400a98f0 2024-03-25T18:26:48.250 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.247+0000 7f3e46a16640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e400b28a0 con 0x7f3e401430c0 2024-03-25T18:26:48.250 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.247+0000 7f3e46a16640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e400b2a30 con 0x7f3e4013f710 2024-03-25T18:26:48.250 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.247+0000 7f3e46a16640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:48.250 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.247+0000 7f3e46a16640 10 monclient(hunting): authenticate will time out at 8220.523456s 2024-03-25T18:26:48.259 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.257+0000 7f3e44fb3640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f3e401430c0 0x7f3e4014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.97:33268/0 (socket says 172.21.15.97:33268) 2024-03-25T18:26:48.259 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.257+0000 7f3e44fb3640 1 -- 172.21.15.97:0/1812161843 learned_addr learned my addr 172.21.15.97:0/1812161843 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:26:48.269 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.267+0000 7f3e3f7fe640 1 -- 172.21.15.97:0/1812161843 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2643853464 0 0) 0x7f3e400b28a0 con 0x7f3e401430c0 2024-03-25T18:26:48.269 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.267+0000 7f3e3f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:26:48.269 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.267+0000 7f3e3f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:48.269 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.267+0000 7f3e3f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:48.269 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.267+0000 7f3e3f7fe640 10 monclient(hunting): my global_id is 4702 2024-03-25T18:26:48.269 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.267+0000 7f3e3f7fe640 1 -- 172.21.15.97:0/1812161843 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e2c003520 con 0x7f3e401430c0 2024-03-25T18:26:48.270 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.269+0000 7f3e3f7fe640 1 -- 172.21.15.97:0/1812161843 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2592294098 0 0) 0x7f3e40142520 con 0x7f3e400a98f0 2024-03-25T18:26:48.270 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.269+0000 7f3e3f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:26:48.270 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.269+0000 7f3e3f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:48.270 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.269+0000 7f3e3f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:48.271 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.269+0000 7f3e3f7fe640 10 monclient(hunting): my global_id is 4692 2024-03-25T18:26:48.271 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.269+0000 7f3e3f7fe640 1 -- 172.21.15.97:0/1812161843 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e400b28a0 con 0x7f3e400a98f0 2024-03-25T18:26:48.279 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.277+0000 7f3e3f7fe640 1 -- 172.21.15.97:0/1812161843 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1385707326 0 0) 0x7f3e2c003520 con 0x7f3e401430c0 2024-03-25T18:26:48.279 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.277+0000 7f3e3f7fe640 1 -- 172.21.15.97:0/1812161843 >> v1:172.21.15.97:6790/0 conn(0x7f3e4013f710 legacy=0x7f3e401429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:48.279 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.277+0000 7f3e3f7fe640 1 -- 172.21.15.97:0/1812161843 >> v1:172.21.15.97:6789/0 conn(0x7f3e400a98f0 legacy=0x7f3e400a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:48.279 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.277+0000 7f3e3f7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:26:48.279 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.277+0000 7f3e3f7fe640 1 monclient: found mon.noname-c 2024-03-25T18:26:48.279 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.277+0000 7f3e3f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:48.279 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.277+0000 7f3e3f7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.98:6789/0 2024-03-25T18:26:48.279 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.277+0000 7f3e3f7fe640 1 -- 172.21.15.97:0/1812161843 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e400b44c0 con 0x7f3e401430c0 2024-03-25T18:26:48.279 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.277+0000 7f3e3f7fe640 10 monclient: _finish_auth 0 2024-03-25T18:26:48.279 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.277+0000 7f3e3f7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:26:48.279 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.277+0000 7f3e3f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:48.279 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.277+0000 7f3e3f7fe640 1 -- 172.21.15.97:0/1812161843 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f3e38001a10 con 0x7f3e401430c0 2024-03-25T18:26:48.280 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.277+0000 7f3e3f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:48.280 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.277+0000 7f3e3f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:26:48.280 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.277+0000 7f3e3f7fe640 10 monclient: dump: 2024-03-25T18:26:48.280 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:26:48.280 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:26:48.280 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:26:48.280 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:26:48.280 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:48.280 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:26:48.280 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:26:48.280 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:26:48.280 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:26:48.280 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:26:48.280 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.277+0000 7f3e46a16640 5 monclient: authenticate success, global_id 4702 2024-03-25T18:26:48.281 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.277+0000 7f3e46a16640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:26:48.284 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.282+0000 7f3e3f7fe640 1 -- 172.21.15.97:0/1812161843 <== mon.1 v1:172.21.15.98:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3e38001b70 con 0x7f3e401430c0 2024-03-25T18:26:48.284 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.282+0000 7f3e3f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:48.284 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.282+0000 7f3e3f7fe640 1 -- 172.21.15.97:0/1812161843 <== mon.1 v1:172.21.15.98:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f3e38001f80 con 0x7f3e401430c0 2024-03-25T18:26:48.284 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.282+0000 7f3e46a16640 10 monclient: get_monmap_and_config success 2024-03-25T18:26:48.284 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.282+0000 7f3e46a16640 4 set_mon_vals no callback set 2024-03-25T18:26:48.284 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.282+0000 7f3e3f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:48.284 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.282+0000 7f3e3f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:26:48.284 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.282+0000 7f3e3f7fe640 10 monclient: dump: 2024-03-25T18:26:48.284 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:26:48.284 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:26:48.284 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:26:48.284 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:26:48.285 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:48.285 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:26:48.285 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:26:48.285 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:26:48.285 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:26:48.285 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:26:48.287 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.283+0000 7f3e46a16640 10 monclient: shutdown 2024-03-25T18:26:48.287 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.283+0000 7f3e46a16640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:48.287 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.283+0000 7f3e46a16640 1 -- 172.21.15.97:0/1812161843 >> v1:172.21.15.98:6789/0 conn(0x7f3e401430c0 legacy=0x7f3e4014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:48.287 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.283+0000 7f3e46a16640 1 -- 172.21.15.97:0/1812161843 shutdown_connections 2024-03-25T18:26:48.288 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.283+0000 7f3e46a16640 1 -- 172.21.15.97:0/1812161843 >> 172.21.15.97:0/1812161843 conn(0x7f3e400b7ea0 msgr2=0x7f3e400b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:48.288 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.283+0000 7f3e46a16640 1 -- 172.21.15.97:0/1812161843 shutdown_connections 2024-03-25T18:26:48.288 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.283+0000 7f3e46a16640 1 -- 172.21.15.97:0/1812161843 wait complete. 2024-03-25T18:26:48.288 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.284+0000 7f3e46a16640 10 monclient: build_initial_monmap 2024-03-25T18:26:48.288 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.284+0000 7f3e46a16640 10 monclient: monmap: 2024-03-25T18:26:48.288 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:26:48.288 INFO:teuthology.misc.health.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:26:48.288 INFO:teuthology.misc.health.smithi097.stderr:last_changed 0.000000 2024-03-25T18:26:48.288 INFO:teuthology.misc.health.smithi097.stderr:created 0.000000 2024-03-25T18:26:48.288 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:48.288 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 1 2024-03-25T18:26:48.288 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:26:48.288 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T18:26:48.289 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-c 2024-03-25T18:26:48.289 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:26:48.289 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.284+0000 7f3e46a16640 1 librados: starting msgr at 2024-03-25T18:26:48.289 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.284+0000 7f3e46a16640 1 librados: starting objecter 2024-03-25T18:26:48.289 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.284+0000 7f3e46a16640 1 Processor -- start 2024-03-25T18:26:48.289 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.284+0000 7f3e46a16640 1 -- start start 2024-03-25T18:26:48.289 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.284+0000 7f3e46a16640 1 librados: setting wanted keys 2024-03-25T18:26:48.289 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.284+0000 7f3e46a16640 1 librados: calling monclient init 2024-03-25T18:26:48.289 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.284+0000 7f3e46a16640 10 monclient: init 2024-03-25T18:26:48.289 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.285+0000 7f3e46a16640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:48.289 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.285+0000 7f3e46a16640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:26:48.289 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.285+0000 7f3e46a16640 10 monclient(hunting): picked mon.noname-a con 0x7f3e400a98f0 addr v1:172.21.15.97:6789/0 2024-03-25T18:26:48.289 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.285+0000 7f3e46a16640 10 monclient(hunting): picked mon.noname-c con 0x7f3e4013f710 addr v1:172.21.15.97:6790/0 2024-03-25T18:26:48.289 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.285+0000 7f3e46a16640 10 monclient(hunting): picked mon.noname-b con 0x7f3e401430c0 addr v1:172.21.15.98:6789/0 2024-03-25T18:26:48.289 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.285+0000 7f3e46a16640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e4013c2b0 con 0x7f3e400a98f0 2024-03-25T18:26:48.290 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.285+0000 7f3e46a16640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e40136f10 con 0x7f3e401430c0 2024-03-25T18:26:48.290 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.285+0000 7f3e46a16640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e401370a0 con 0x7f3e4013f710 2024-03-25T18:26:48.290 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.285+0000 7f3e46a16640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:48.290 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.285+0000 7f3e46a16640 10 monclient(hunting): authenticate will time out at 8220.561360s 2024-03-25T18:26:48.300 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.299+0000 7f3e3ffff640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f3e400a98f0 0x7f3e401d7590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:47982/0 (socket says 172.21.15.97:47982) 2024-03-25T18:26:48.300 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.299+0000 7f3e3ffff640 1 -- 172.21.15.97:0/3636778596 learned_addr learned my addr 172.21.15.97:0/3636778596 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:26:48.302 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.301+0000 7f3e3d7fa640 10 client.?.objecter ms_handle_connect 0x7f3e400a98f0 2024-03-25T18:26:48.302 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.301+0000 7f3e3d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:48.306 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.304+0000 7f3e3d7fa640 10 client.?.objecter ms_handle_connect 0x7f3e401430c0 2024-03-25T18:26:48.306 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.304+0000 7f3e3d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:48.309 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.307+0000 7f3e3d7fa640 10 client.?.objecter ms_handle_connect 0x7f3e4013f710 2024-03-25T18:26:48.309 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.307+0000 7f3e3d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:48.314 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.312+0000 7f3e3d7fa640 1 -- 172.21.15.97:0/3636778596 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1085118417 0 0) 0x7f3e4013c2b0 con 0x7f3e400a98f0 2024-03-25T18:26:48.314 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.312+0000 7f3e3d7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:26:48.314 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.312+0000 7f3e3d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:48.314 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.312+0000 7f3e3d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:48.314 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.312+0000 7f3e3d7fa640 10 monclient(hunting): my global_id is 4695 2024-03-25T18:26:48.315 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.312+0000 7f3e3d7fa640 1 -- 172.21.15.97:0/3636778596 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e20004eb0 con 0x7f3e400a98f0 2024-03-25T18:26:48.315 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.314+0000 7f3e3d7fa640 1 -- 172.21.15.97:0/3636778596 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 18725427 0 0) 0x7f3e40136f10 con 0x7f3e401430c0 2024-03-25T18:26:48.315 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.314+0000 7f3e3d7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:26:48.315 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.314+0000 7f3e3d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:48.315 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.314+0000 7f3e3d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:48.317 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.314+0000 7f3e3d7fa640 10 monclient(hunting): my global_id is 4705 2024-03-25T18:26:48.317 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.314+0000 7f3e3d7fa640 1 -- 172.21.15.97:0/3636778596 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e4013c2b0 con 0x7f3e401430c0 2024-03-25T18:26:48.318 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.317+0000 7f3e3d7fa640 1 -- 172.21.15.97:0/3636778596 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4075187187 0 0) 0x7f3e401370a0 con 0x7f3e4013f710 2024-03-25T18:26:48.318 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.317+0000 7f3e3d7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:26:48.318 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.317+0000 7f3e3d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:48.318 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.317+0000 7f3e3d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:48.318 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.317+0000 7f3e3d7fa640 10 monclient(hunting): my global_id is 4679 2024-03-25T18:26:48.319 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.317+0000 7f3e3d7fa640 1 -- 172.21.15.97:0/3636778596 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e40136f10 con 0x7f3e4013f710 2024-03-25T18:26:48.331 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.328+0000 7f3e3d7fa640 1 -- 172.21.15.97:0/3636778596 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3078052340 0 0) 0x7f3e20004eb0 con 0x7f3e400a98f0 2024-03-25T18:26:48.331 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.329+0000 7f3e3d7fa640 1 -- 172.21.15.97:0/3636778596 --> v1:172.21.15.97:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3e401370a0 con 0x7f3e400a98f0 2024-03-25T18:26:48.331 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.329+0000 7f3e3d7fa640 1 -- 172.21.15.97:0/3636778596 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f3e30001a10 con 0x7f3e400a98f0 2024-03-25T18:26:48.331 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.329+0000 7f3e3d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:48.331 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.329+0000 7f3e3d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:26:48.331 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.329+0000 7f3e3d7fa640 10 monclient(hunting): dump: 2024-03-25T18:26:48.331 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:26:48.331 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:26:48.331 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:26:48.331 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:26:48.331 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:48.331 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:26:48.332 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:26:48.332 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:26:48.332 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:26:48.332 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:26:48.332 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.329+0000 7f3e3d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:26:48.338 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.335+0000 7f3e3d7fa640 1 -- 172.21.15.97:0/3636778596 <== mon.0 v1:172.21.15.97:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4289236628 0 0) 0x7f3e401370a0 con 0x7f3e400a98f0 2024-03-25T18:26:48.338 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.335+0000 7f3e3d7fa640 1 -- 172.21.15.97:0/3636778596 >> v1:172.21.15.97:6790/0 conn(0x7f3e4013f710 legacy=0x7f3e401377d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:48.338 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.335+0000 7f3e3d7fa640 1 -- 172.21.15.97:0/3636778596 >> v1:172.21.15.98:6789/0 conn(0x7f3e401430c0 legacy=0x7f3e40133000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:48.338 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.335+0000 7f3e3d7fa640 10 monclient: _finish_hunting 0 2024-03-25T18:26:48.338 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.335+0000 7f3e3d7fa640 1 monclient: found mon.a 2024-03-25T18:26:48.338 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.335+0000 7f3e3d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:48.338 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.335+0000 7f3e3d7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:26:48.338 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.335+0000 7f3e3d7fa640 1 -- 172.21.15.97:0/3636778596 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e40137230 con 0x7f3e400a98f0 2024-03-25T18:26:48.338 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.335+0000 7f3e3d7fa640 10 monclient: _finish_auth 0 2024-03-25T18:26:48.338 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.335+0000 7f3e3d7fa640 10 monclient: _check_auth_tickets 2024-03-25T18:26:48.338 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.335+0000 7f3e3d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:48.339 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.336+0000 7f3e46a16640 5 monclient: authenticate success, global_id 4695 2024-03-25T18:26:48.339 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.336+0000 7f3e46a16640 10 monclient: _renew_subs 2024-03-25T18:26:48.339 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.336+0000 7f3e46a16640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:26:48.339 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.336+0000 7f3e46a16640 1 -- 172.21.15.97:0/3636778596 --> v1:172.21.15.97:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e4013b620 con 0x7f3e400a98f0 2024-03-25T18:26:48.339 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.336+0000 7f3e46a16640 10 client.4695.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:26:48.339 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.336+0000 7f3e46a16640 10 monclient: _renew_subs 2024-03-25T18:26:48.339 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.336+0000 7f3e46a16640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:26:48.339 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.336+0000 7f3e46a16640 1 -- 172.21.15.97:0/3636778596 --> v1:172.21.15.97:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e401338f0 con 0x7f3e400a98f0 2024-03-25T18:26:48.339 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.336+0000 7f3e46a16640 1 librados: init done 2024-03-25T18:26:48.340 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.337+0000 7f3e46a16640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:48.340 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.337+0000 7f3e46a16640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:48.340 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.337+0000 7f3e46a16640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:26:48.340 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.337+0000 7f3e46a16640 1 -- 172.21.15.97:0/3636778596 --> v1:172.21.15.97:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e4013b9c0 con 0x7f3e400a98f0 2024-03-25T18:26:48.350 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.349+0000 7f3e3d7fa640 1 -- 172.21.15.97:0/3636778596 <== mon.0 v1:172.21.15.97:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3e30001c50 con 0x7f3e400a98f0 2024-03-25T18:26:48.350 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.349+0000 7f3e3d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:48.351 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.349+0000 7f3e3effd640 4 set_mon_vals no callback set 2024-03-25T18:26:48.639 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.637+0000 7f3e3d7fa640 1 -- 172.21.15.97:0/3636778596 <== mon.0 v1:172.21.15.97:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f3e300038c0 con 0x7f3e400a98f0 2024-03-25T18:26:48.639 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.637+0000 7f3e3d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:48.639 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.637+0000 7f3e3d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:26:48.639 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.637+0000 7f3e3d7fa640 10 monclient: dump: 2024-03-25T18:26:48.639 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:26:48.639 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:26:48.640 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:26:48.640 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:26:48.640 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:48.640 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:26:48.640 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:26:48.640 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:26:48.640 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:26:48.640 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:26:48.641 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.638+0000 7f3e3d7fa640 1 -- 172.21.15.97:0/3636778596 <== mon.0 v1:172.21.15.97:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2377622137 0 0) 0x7f3e30003ae0 con 0x7f3e400a98f0 2024-03-25T18:26:48.641 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.639+0000 7f3e37fff640 10 monclient: get_auth_request con 0x7f3e2006dda0 auth_method 0 2024-03-25T18:26:48.641 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.639+0000 7f3e3d7fa640 10 client.4695.objecter ms_handle_connect 0x7f3e2006dda0 2024-03-25T18:26:48.646 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.644+0000 7f3e3d7fa640 1 -- 172.21.15.97:0/3636778596 <== mon.0 v1:172.21.15.97:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3078631805 0 0) 0x7f3e300547c0 con 0x7f3e400a98f0 2024-03-25T18:26:48.647 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.645+0000 7f3e3d7fa640 10 client.4695.objecter ms_dispatch 0x7f3e40000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:26:48.647 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.645+0000 7f3e3d7fa640 3 client.4695.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:26:48.647 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.645+0000 7f3e3d7fa640 3 client.4695.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:26:48.647 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.645+0000 7f3e3d7fa640 20 client.4695.objecter dump_active .. 0 homeless 2024-03-25T18:26:48.647 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.645+0000 7f3e3d7fa640 1 -- 172.21.15.97:0/3636778596 <== mon.0 v1:172.21.15.97:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3e300502d0 con 0x7f3e400a98f0 2024-03-25T18:26:48.647 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.645+0000 7f3e3d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:48.647 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.645+0000 7f3e3d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:26:48.868 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.866+0000 7f3e46a16640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:48.868 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.866+0000 7f3e46a16640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:48.868 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.866+0000 7f3e46a16640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:26:48.868 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.866+0000 7f3e46a16640 1 -- 172.21.15.97:0/3636778596 --> v1:172.21.15.97:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3e400627b0 con 0x7f3e400a98f0 2024-03-25T18:26:48.914 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.912+0000 7f3e3d7fa640 1 -- 172.21.15.97:0/3636778596 <== mon.0 v1:172.21.15.97:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 337131954) 0x7f3e30052200 con 0x7f3e400a98f0 2024-03-25T18:26:48.914 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.912+0000 7f3e3d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:48.914 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.912+0000 7f3e3d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:26:48.915 INFO:teuthology.misc.health.smithi097.stdout: 2024-03-25T18:26:48.915 INFO:teuthology.misc.health.smithi097.stdout:{"epoch":11,"fsid":"4c3d472c-79b8-486b-9e3d-2738d4306754","created":"2024-03-25T18:16:14.737309+0000","modified":"2024-03-25T18:26:18.750062+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:18.750062+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cb5152b9-262f-45c3-94b8-18d0e81df558","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"6daf8793-0838-473d-a7d6-9a0d2894c75d","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"136f1493-3b9c-4f43-a2e7-641482d69dae","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"397d538c-4f64-4daf-9e8b-662d09f91327","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4f1df08-663f-4bda-93f8-3878ab7c24b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c4b18d70-a3ce-4b15-9b70-eae5525bd761","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8dd4033a-de0e-4f8a-96dc-59f3c1f825b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0077722e-ffa6-4a3d-9891-929d9869b286","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:26:48.917 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.915+0000 7f3e46a16640 10 librados: watch_flush enter 2024-03-25T18:26:48.917 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.915+0000 7f3e46a16640 10 librados: watch_flush exit 2024-03-25T18:26:48.917 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.915+0000 7f3e46a16640 20 client.4695.objecter shutdown clearing up homeless session... 2024-03-25T18:26:48.917 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.915+0000 7f3e46a16640 10 client.4695.objecter successfully canceled tick 2024-03-25T18:26:48.918 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.915+0000 7f3e46a16640 1 -- 172.21.15.97:0/3636778596 >> v1:172.21.15.98:6800/800387573 conn(0x7f3e2006dda0 legacy=0x7f3e20070260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:48.918 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.915+0000 7f3e46a16640 10 monclient: shutdown 2024-03-25T18:26:48.918 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.915+0000 7f3e46a16640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:48.918 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.915+0000 7f3e46a16640 1 -- 172.21.15.97:0/3636778596 >> v1:172.21.15.97:6789/0 conn(0x7f3e400a98f0 legacy=0x7f3e401d7590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:48.918 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.915+0000 7f3e46a16640 1 -- 172.21.15.97:0/3636778596 shutdown_connections 2024-03-25T18:26:48.918 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.915+0000 7f3e46a16640 1 -- 172.21.15.97:0/3636778596 >> 172.21.15.97:0/3636778596 conn(0x7f3e400b7ea0 msgr2=0x7f3e40074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:48.918 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.915+0000 7f3e46a16640 1 -- 172.21.15.97:0/3636778596 shutdown_connections 2024-03-25T18:26:48.918 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.915+0000 7f3e46a16640 1 -- 172.21.15.97:0/3636778596 wait complete. 2024-03-25T18:26:48.918 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:48.915+0000 7f3e46a16640 1 librados: shutdown 2024-03-25T18:26:48.928 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:26:55.735 DEBUG:teuthology.orchestra.run.smithi097:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:26:55.847 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.845+0000 7faf8b4c5640 10 monclient: get_monmap_and_config 2024-03-25T18:26:55.847 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.845+0000 7faf8b4c5640 10 monclient: build_initial_monmap 2024-03-25T18:26:55.847 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.845+0000 7faf8b4c5640 10 monclient: monmap: 2024-03-25T18:26:55.847 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:26:55.847 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:26:55.847 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:26:55.846912+0000 2024-03-25T18:26:55.847 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:26:55.846912+0000 2024-03-25T18:26:55.848 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:55.848 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:26:55.848 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:26:55.848 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T18:26:55.848 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-b 2024-03-25T18:26:55.848 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:26:55.849 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.846+0000 7faf8b4c5640 1 Processor -- start 2024-03-25T18:26:55.849 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.846+0000 7faf8b4c5640 1 -- start start 2024-03-25T18:26:55.849 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.846+0000 7faf8b4c5640 10 monclient: init 2024-03-25T18:26:55.849 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.847+0000 7faf8b4c5640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:55.849 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.847+0000 7faf8b4c5640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:26:55.849 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.847+0000 7faf8b4c5640 10 monclient(hunting): picked mon.noname-c con 0x7faf840a98f0 addr v1:172.21.15.98:6789/0 2024-03-25T18:26:55.849 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.847+0000 7faf8b4c5640 10 monclient(hunting): picked mon.noname-a con 0x7faf8413f710 addr v1:172.21.15.97:6789/0 2024-03-25T18:26:55.849 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.847+0000 7faf8b4c5640 10 monclient(hunting): picked mon.noname-b con 0x7faf841430c0 addr v1:172.21.15.97:6790/0 2024-03-25T18:26:55.849 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.847+0000 7faf8b4c5640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf84142520 con 0x7faf8413f710 2024-03-25T18:26:55.849 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.847+0000 7faf8b4c5640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf840b28a0 con 0x7faf840a98f0 2024-03-25T18:26:55.849 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.847+0000 7faf8b4c5640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf840b2a30 con 0x7faf841430c0 2024-03-25T18:26:55.850 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.847+0000 7faf8b4c5640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:55.850 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.847+0000 7faf8b4c5640 10 monclient(hunting): authenticate will time out at 8228.123208s 2024-03-25T18:26:55.852 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.850+0000 7faf89a62640 1 --1- >> v1:172.21.15.97:6790/0 conn(0x7faf841430c0 0x7faf8414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6790/0 says I am v1:172.21.15.97:46622/0 (socket says 172.21.15.97:46622) 2024-03-25T18:26:55.852 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.850+0000 7faf89a62640 1 -- 172.21.15.97:0/1687018422 learned_addr learned my addr 172.21.15.97:0/1687018422 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:26:55.876 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.874+0000 7faf7bfff640 1 -- 172.21.15.97:0/1687018422 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 900013808 0 0) 0x7faf84142520 con 0x7faf8413f710 2024-03-25T18:26:55.876 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.874+0000 7faf7bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:26:55.877 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.874+0000 7faf7bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:55.877 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.874+0000 7faf7bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:55.877 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.874+0000 7faf7bfff640 10 monclient(hunting): my global_id is 4698 2024-03-25T18:26:55.877 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.874+0000 7faf7bfff640 1 -- 172.21.15.97:0/1687018422 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf70003520 con 0x7faf8413f710 2024-03-25T18:26:55.877 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.874+0000 7faf7bfff640 1 -- 172.21.15.97:0/1687018422 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3258042807 0 0) 0x7faf840b2a30 con 0x7faf841430c0 2024-03-25T18:26:55.877 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.874+0000 7faf7bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:26:55.877 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.874+0000 7faf7bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:55.877 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.874+0000 7faf7bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:55.877 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.874+0000 7faf7bfff640 10 monclient(hunting): my global_id is 4682 2024-03-25T18:26:55.878 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.874+0000 7faf7bfff640 1 -- 172.21.15.97:0/1687018422 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf84142520 con 0x7faf841430c0 2024-03-25T18:26:55.878 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.874+0000 7faf7bfff640 1 -- 172.21.15.97:0/1687018422 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2455172753 0 0) 0x7faf840b28a0 con 0x7faf840a98f0 2024-03-25T18:26:55.878 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.874+0000 7faf7bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:26:55.878 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.874+0000 7faf7bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:55.878 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.874+0000 7faf7bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:55.878 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.874+0000 7faf7bfff640 10 monclient(hunting): my global_id is 4708 2024-03-25T18:26:55.878 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.874+0000 7faf7bfff640 1 -- 172.21.15.97:0/1687018422 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf840b2a30 con 0x7faf840a98f0 2024-03-25T18:26:55.882 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.880+0000 7faf7bfff640 1 -- 172.21.15.97:0/1687018422 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1757148405 0 0) 0x7faf70003520 con 0x7faf8413f710 2024-03-25T18:26:55.882 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.880+0000 7faf7bfff640 1 -- 172.21.15.97:0/1687018422 >> v1:172.21.15.97:6790/0 conn(0x7faf841430c0 legacy=0x7faf8414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:55.882 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.880+0000 7faf7bfff640 1 -- 172.21.15.97:0/1687018422 >> v1:172.21.15.98:6789/0 conn(0x7faf840a98f0 legacy=0x7faf840a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:55.882 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.880+0000 7faf7bfff640 10 monclient: _finish_hunting 0 2024-03-25T18:26:55.883 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.880+0000 7faf7bfff640 1 monclient: found mon.noname-a 2024-03-25T18:26:55.883 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.880+0000 7faf7bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:55.883 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.880+0000 7faf7bfff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.97:6789/0 2024-03-25T18:26:55.883 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.880+0000 7faf7bfff640 1 -- 172.21.15.97:0/1687018422 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf840b44c0 con 0x7faf8413f710 2024-03-25T18:26:55.883 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.880+0000 7faf7bfff640 10 monclient: _finish_auth 0 2024-03-25T18:26:55.883 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.880+0000 7faf7bfff640 10 monclient: _check_auth_tickets 2024-03-25T18:26:55.883 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.880+0000 7faf7bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:55.883 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.880+0000 7faf7bfff640 1 -- 172.21.15.97:0/1687018422 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7faf74001690 con 0x7faf8413f710 2024-03-25T18:26:55.883 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.880+0000 7faf7bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:55.883 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.880+0000 7faf7bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:26:55.883 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.880+0000 7faf7bfff640 10 monclient: dump: 2024-03-25T18:26:55.883 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:26:55.883 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:26:55.884 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:26:55.884 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:26:55.884 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:55.884 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:26:55.884 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:26:55.884 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:26:55.884 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:26:55.884 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:26:55.884 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.880+0000 7faf8b4c5640 5 monclient: authenticate success, global_id 4698 2024-03-25T18:26:55.884 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.880+0000 7faf8b4c5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:26:55.892 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.890+0000 7faf7bfff640 1 -- 172.21.15.97:0/1687018422 <== mon.0 v1:172.21.15.97:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faf740017f0 con 0x7faf8413f710 2024-03-25T18:26:55.892 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.890+0000 7faf7bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:55.892 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.890+0000 7faf7bfff640 1 -- 172.21.15.97:0/1687018422 <== mon.0 v1:172.21.15.97:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7faf74001c00 con 0x7faf8413f710 2024-03-25T18:26:55.892 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.890+0000 7faf8b4c5640 10 monclient: get_monmap_and_config success 2024-03-25T18:26:55.892 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.890+0000 7faf8b4c5640 4 set_mon_vals no callback set 2024-03-25T18:26:55.892 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.890+0000 7faf7bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:55.892 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.890+0000 7faf7bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:26:55.892 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.890+0000 7faf7bfff640 10 monclient: dump: 2024-03-25T18:26:55.893 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:26:55.893 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:26:55.893 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:26:55.893 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:26:55.893 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:55.893 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:26:55.893 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:26:55.893 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:26:55.893 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:26:55.893 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:26:55.895 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.891+0000 7faf8b4c5640 10 monclient: shutdown 2024-03-25T18:26:55.896 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.891+0000 7faf8b4c5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:55.896 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.891+0000 7faf8b4c5640 1 -- 172.21.15.97:0/1687018422 >> v1:172.21.15.97:6789/0 conn(0x7faf8413f710 legacy=0x7faf841429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:55.896 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.891+0000 7faf8b4c5640 1 -- 172.21.15.97:0/1687018422 shutdown_connections 2024-03-25T18:26:55.896 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.891+0000 7faf8b4c5640 1 -- 172.21.15.97:0/1687018422 >> 172.21.15.97:0/1687018422 conn(0x7faf840b7ea0 msgr2=0x7faf840b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:55.896 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.891+0000 7faf8b4c5640 1 -- 172.21.15.97:0/1687018422 shutdown_connections 2024-03-25T18:26:55.896 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.891+0000 7faf8b4c5640 1 -- 172.21.15.97:0/1687018422 wait complete. 2024-03-25T18:26:55.896 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.892+0000 7faf8b4c5640 10 monclient: build_initial_monmap 2024-03-25T18:26:55.896 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.892+0000 7faf8b4c5640 10 monclient: monmap: 2024-03-25T18:26:55.896 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:26:55.896 INFO:teuthology.misc.health.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:26:55.896 INFO:teuthology.misc.health.smithi097.stderr:last_changed 0.000000 2024-03-25T18:26:55.896 INFO:teuthology.misc.health.smithi097.stderr:created 0.000000 2024-03-25T18:26:55.896 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:55.896 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 1 2024-03-25T18:26:55.897 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:26:55.897 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T18:26:55.897 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-c 2024-03-25T18:26:55.897 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:26:55.897 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.892+0000 7faf8b4c5640 1 librados: starting msgr at 2024-03-25T18:26:55.897 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.892+0000 7faf8b4c5640 1 librados: starting objecter 2024-03-25T18:26:55.897 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.893+0000 7faf8b4c5640 1 Processor -- start 2024-03-25T18:26:55.897 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.893+0000 7faf8b4c5640 1 -- start start 2024-03-25T18:26:55.897 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.893+0000 7faf8b4c5640 1 librados: setting wanted keys 2024-03-25T18:26:55.897 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.893+0000 7faf8b4c5640 1 librados: calling monclient init 2024-03-25T18:26:55.897 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.893+0000 7faf8b4c5640 10 monclient: init 2024-03-25T18:26:55.897 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.893+0000 7faf8b4c5640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:55.897 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.893+0000 7faf8b4c5640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:26:55.898 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.893+0000 7faf8b4c5640 10 monclient(hunting): picked mon.noname-a con 0x7faf840a98f0 addr v1:172.21.15.97:6789/0 2024-03-25T18:26:55.898 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.893+0000 7faf8b4c5640 10 monclient(hunting): picked mon.noname-c con 0x7faf8413f710 addr v1:172.21.15.97:6790/0 2024-03-25T18:26:55.898 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.893+0000 7faf8b4c5640 10 monclient(hunting): picked mon.noname-b con 0x7faf841430c0 addr v1:172.21.15.98:6789/0 2024-03-25T18:26:55.898 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.893+0000 7faf8b4c5640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf840b79f0 con 0x7faf840a98f0 2024-03-25T18:26:55.898 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.893+0000 7faf8b4c5640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf841350e0 con 0x7faf841430c0 2024-03-25T18:26:55.898 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.893+0000 7faf8b4c5640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf84135270 con 0x7faf8413f710 2024-03-25T18:26:55.901 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.893+0000 7faf8b4c5640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:55.901 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.893+0000 7faf8b4c5640 10 monclient(hunting): authenticate will time out at 8228.169676s 2024-03-25T18:26:55.904 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.902+0000 7faf88a60640 1 --1- >> v1:172.21.15.97:6790/0 conn(0x7faf8413f710 0x7faf841d7d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.97:6790/0 says I am v1:172.21.15.97:46660/0 (socket says 172.21.15.97:46660) 2024-03-25T18:26:55.904 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.902+0000 7faf88a60640 1 -- 172.21.15.97:0/2915148810 learned_addr learned my addr 172.21.15.97:0/2915148810 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:26:55.904 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.903+0000 7faf79ffb640 10 client.?.objecter ms_handle_connect 0x7faf8413f710 2024-03-25T18:26:55.904 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.903+0000 7faf79ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:55.905 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.903+0000 7faf79ffb640 10 client.?.objecter ms_handle_connect 0x7faf841430c0 2024-03-25T18:26:55.905 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.903+0000 7faf79ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:55.913 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.912+0000 7faf79ffb640 10 client.?.objecter ms_handle_connect 0x7faf840a98f0 2024-03-25T18:26:55.914 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.912+0000 7faf79ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:55.920 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.918+0000 7faf79ffb640 1 -- 172.21.15.97:0/2915148810 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1990029727 0 0) 0x7faf841350e0 con 0x7faf841430c0 2024-03-25T18:26:55.920 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.918+0000 7faf79ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:26:55.920 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.918+0000 7faf79ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:55.920 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.918+0000 7faf79ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:55.920 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.918+0000 7faf79ffb640 10 monclient(hunting): my global_id is 4711 2024-03-25T18:26:55.920 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.918+0000 7faf79ffb640 1 -- 172.21.15.97:0/2915148810 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf68004eb0 con 0x7faf841430c0 2024-03-25T18:26:55.929 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.926+0000 7faf79ffb640 1 -- 172.21.15.97:0/2915148810 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 412046222 0 0) 0x7faf840b79f0 con 0x7faf840a98f0 2024-03-25T18:26:55.929 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.926+0000 7faf79ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:26:55.929 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.926+0000 7faf79ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:55.929 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.926+0000 7faf79ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:55.929 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.926+0000 7faf79ffb640 10 monclient(hunting): my global_id is 4701 2024-03-25T18:26:55.929 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.926+0000 7faf79ffb640 1 -- 172.21.15.97:0/2915148810 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf841350e0 con 0x7faf840a98f0 2024-03-25T18:26:55.930 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.929+0000 7faf79ffb640 1 -- 172.21.15.97:0/2915148810 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3911875565 0 0) 0x7faf84135270 con 0x7faf8413f710 2024-03-25T18:26:55.930 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.929+0000 7faf79ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:26:55.931 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.929+0000 7faf79ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:55.931 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.929+0000 7faf79ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:55.931 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.929+0000 7faf79ffb640 10 monclient(hunting): my global_id is 4685 2024-03-25T18:26:55.931 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.929+0000 7faf79ffb640 1 -- 172.21.15.97:0/2915148810 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf840b79f0 con 0x7faf8413f710 2024-03-25T18:26:55.932 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.931+0000 7faf79ffb640 1 -- 172.21.15.97:0/2915148810 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2686707191 0 0) 0x7faf68004eb0 con 0x7faf841430c0 2024-03-25T18:26:55.933 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.931+0000 7faf79ffb640 1 -- 172.21.15.97:0/2915148810 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faf84135270 con 0x7faf841430c0 2024-03-25T18:26:55.933 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.931+0000 7faf79ffb640 1 -- 172.21.15.97:0/2915148810 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7faf7c001bd0 con 0x7faf841430c0 2024-03-25T18:26:55.933 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.931+0000 7faf79ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:55.933 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.931+0000 7faf79ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:26:55.933 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.931+0000 7faf79ffb640 10 monclient(hunting): dump: 2024-03-25T18:26:55.933 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:26:55.933 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:26:55.933 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:26:55.933 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:26:55.933 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:55.933 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:26:55.933 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:26:55.933 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:26:55.933 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:26:55.934 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:26:55.934 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.931+0000 7faf79ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:26:55.940 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.938+0000 7faf79ffb640 1 -- 172.21.15.97:0/2915148810 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 985850165 0 0) 0x7faf841350e0 con 0x7faf840a98f0 2024-03-25T18:26:55.940 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.938+0000 7faf79ffb640 1 -- 172.21.15.97:0/2915148810 --> v1:172.21.15.97:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faf68004eb0 con 0x7faf840a98f0 2024-03-25T18:26:55.940 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.938+0000 7faf79ffb640 1 -- 172.21.15.97:0/2915148810 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7faf80001870 con 0x7faf840a98f0 2024-03-25T18:26:55.940 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.938+0000 7faf79ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:55.940 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.938+0000 7faf79ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:26:55.940 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.938+0000 7faf79ffb640 10 monclient(hunting): dump: 2024-03-25T18:26:55.940 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:26:55.940 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:26:55.940 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:26:55.940 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:26:55.941 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:55.941 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:26:55.941 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:26:55.941 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:26:55.941 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:26:55.941 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:26:55.945 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.942+0000 7faf79ffb640 1 -- 172.21.15.97:0/2915148810 <== mon.1 v1:172.21.15.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 642864443 0 0) 0x7faf84135270 con 0x7faf841430c0 2024-03-25T18:26:55.945 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.942+0000 7faf79ffb640 1 -- 172.21.15.97:0/2915148810 >> v1:172.21.15.97:6790/0 conn(0x7faf8413f710 legacy=0x7faf841d7d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:55.945 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.942+0000 7faf79ffb640 1 -- 172.21.15.97:0/2915148810 >> v1:172.21.15.97:6789/0 conn(0x7faf840a98f0 legacy=0x7faf841d7670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:55.945 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.942+0000 7faf79ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:26:55.945 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.942+0000 7faf79ffb640 1 monclient: found mon.b 2024-03-25T18:26:55.945 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.942+0000 7faf79ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:55.945 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.942+0000 7faf79ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:26:55.945 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.942+0000 7faf79ffb640 1 -- 172.21.15.97:0/2915148810 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf84135400 con 0x7faf841430c0 2024-03-25T18:26:55.945 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.942+0000 7faf79ffb640 10 monclient: _finish_auth 0 2024-03-25T18:26:55.945 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.942+0000 7faf79ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:26:55.945 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.942+0000 7faf79ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:55.945 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.942+0000 7faf8b4c5640 5 monclient: authenticate success, global_id 4711 2024-03-25T18:26:55.945 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.942+0000 7faf8b4c5640 10 monclient: _renew_subs 2024-03-25T18:26:55.946 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.942+0000 7faf8b4c5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:26:55.946 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.942+0000 7faf8b4c5640 1 -- 172.21.15.97:0/2915148810 --> v1:172.21.15.98:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf8413b630 con 0x7faf841430c0 2024-03-25T18:26:55.946 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.942+0000 7faf8b4c5640 10 client.4711.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:26:55.946 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.942+0000 7faf8b4c5640 10 monclient: _renew_subs 2024-03-25T18:26:55.946 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.942+0000 7faf8b4c5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:26:55.946 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.942+0000 7faf8b4c5640 1 -- 172.21.15.97:0/2915148810 --> v1:172.21.15.98:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7faf8413b9d0 con 0x7faf841430c0 2024-03-25T18:26:55.946 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.942+0000 7faf8b4c5640 1 librados: init done 2024-03-25T18:26:55.947 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.944+0000 7faf8b4c5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:55.947 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.944+0000 7faf8b4c5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:55.947 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.944+0000 7faf8b4c5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:26:55.947 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:55.944+0000 7faf8b4c5640 1 -- 172.21.15.97:0/2915148810 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf840627b0 con 0x7faf841430c0 2024-03-25T18:26:56.162 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:56.160+0000 7faf79ffb640 1 -- 172.21.15.97:0/2915148810 <== mon.1 v1:172.21.15.98:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faf7c0020e0 con 0x7faf841430c0 2024-03-25T18:26:56.162 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:56.160+0000 7faf79ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:56.162 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:56.160+0000 7faf7b7fe640 4 set_mon_vals no callback set 2024-03-25T18:26:56.216 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:56.214+0000 7faf79ffb640 1 -- 172.21.15.97:0/2915148810 <== mon.1 v1:172.21.15.98:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7faf7c004b60 con 0x7faf841430c0 2024-03-25T18:26:56.216 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:56.214+0000 7faf79ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:56.216 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:56.214+0000 7faf79ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:26:56.216 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:56.214+0000 7faf79ffb640 10 monclient: dump: 2024-03-25T18:26:56.216 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:26:56.216 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:26:56.216 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:26:56.216 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:26:56.216 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:56.216 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:26:56.217 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:26:56.217 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:26:56.217 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:26:56.217 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:26:56.217 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:56.215+0000 7faf79ffb640 1 -- 172.21.15.97:0/2915148810 <== mon.1 v1:172.21.15.98:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2377622137 0 0) 0x7faf7c01abf0 con 0x7faf841430c0 2024-03-25T18:26:56.217 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:56.215+0000 7faf89261640 10 monclient: get_auth_request con 0x7faf6806df20 auth_method 0 2024-03-25T18:26:56.218 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:56.215+0000 7faf79ffb640 10 client.4711.objecter ms_handle_connect 0x7faf6806df20 2024-03-25T18:26:56.219 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:56.217+0000 7faf79ffb640 1 -- 172.21.15.97:0/2915148810 <== mon.1 v1:172.21.15.98:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3078631805 0 0) 0x7faf7c0549b0 con 0x7faf841430c0 2024-03-25T18:26:56.219 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:56.217+0000 7faf79ffb640 10 client.4711.objecter ms_dispatch 0x7faf84000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:26:56.219 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:56.217+0000 7faf79ffb640 3 client.4711.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:26:56.219 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:56.217+0000 7faf79ffb640 3 client.4711.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:26:56.219 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:56.217+0000 7faf79ffb640 20 client.4711.objecter dump_active .. 0 homeless 2024-03-25T18:26:56.223 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:56.221+0000 7faf79ffb640 1 -- 172.21.15.97:0/2915148810 <== mon.1 v1:172.21.15.98:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7faf7c0502f0 con 0x7faf841430c0 2024-03-25T18:26:56.223 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:56.221+0000 7faf79ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:56.223 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:56.221+0000 7faf79ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:26:56.452 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:56.450+0000 7faf8b4c5640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:56.452 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:56.450+0000 7faf8b4c5640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:56.452 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:56.450+0000 7faf8b4c5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:26:56.452 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:56.450+0000 7faf8b4c5640 1 -- 172.21.15.97:0/2915148810 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faf8413e270 con 0x7faf841430c0 2024-03-25T18:26:56.495 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:56.494+0000 7faf79ffb640 1 -- 172.21.15.97:0/2915148810 <== mon.1 v1:172.21.15.98:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 337131954) 0x7faf7c0523f0 con 0x7faf841430c0 2024-03-25T18:26:56.495 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:56.494+0000 7faf79ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:56.496 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:56.494+0000 7faf79ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:26:56.496 INFO:teuthology.misc.health.smithi097.stdout: 2024-03-25T18:26:56.496 INFO:teuthology.misc.health.smithi097.stdout:{"epoch":11,"fsid":"4c3d472c-79b8-486b-9e3d-2738d4306754","created":"2024-03-25T18:16:14.737309+0000","modified":"2024-03-25T18:26:18.750062+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:18.750062+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cb5152b9-262f-45c3-94b8-18d0e81df558","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"6daf8793-0838-473d-a7d6-9a0d2894c75d","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"136f1493-3b9c-4f43-a2e7-641482d69dae","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"397d538c-4f64-4daf-9e8b-662d09f91327","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4f1df08-663f-4bda-93f8-3878ab7c24b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c4b18d70-a3ce-4b15-9b70-eae5525bd761","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8dd4033a-de0e-4f8a-96dc-59f3c1f825b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0077722e-ffa6-4a3d-9891-929d9869b286","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:26:56.498 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:56.496+0000 7faf8b4c5640 10 librados: watch_flush enter 2024-03-25T18:26:56.498 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:56.496+0000 7faf8b4c5640 10 librados: watch_flush exit 2024-03-25T18:26:56.498 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:56.496+0000 7faf8b4c5640 20 client.4711.objecter shutdown clearing up homeless session... 2024-03-25T18:26:56.498 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:56.496+0000 7faf8b4c5640 10 client.4711.objecter successfully canceled tick 2024-03-25T18:26:56.499 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:56.496+0000 7faf8b4c5640 1 -- 172.21.15.97:0/2915148810 >> v1:172.21.15.98:6800/800387573 conn(0x7faf6806df20 legacy=0x7faf680703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:56.499 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:56.496+0000 7faf8b4c5640 10 monclient: shutdown 2024-03-25T18:26:56.499 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:56.496+0000 7faf8b4c5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:56.499 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:56.496+0000 7faf8b4c5640 1 -- 172.21.15.97:0/2915148810 >> v1:172.21.15.98:6789/0 conn(0x7faf841430c0 legacy=0x7faf841e20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:56.499 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:56.496+0000 7faf8b4c5640 1 -- 172.21.15.97:0/2915148810 shutdown_connections 2024-03-25T18:26:56.499 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:56.496+0000 7faf8b4c5640 1 -- 172.21.15.97:0/2915148810 >> 172.21.15.97:0/2915148810 conn(0x7faf840b7ea0 msgr2=0x7faf8413ecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:56.499 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:56.496+0000 7faf8b4c5640 1 -- 172.21.15.97:0/2915148810 shutdown_connections 2024-03-25T18:26:56.499 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:56.496+0000 7faf8b4c5640 1 -- 172.21.15.97:0/2915148810 wait complete. 2024-03-25T18:26:56.499 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:26:56.496+0000 7faf8b4c5640 1 librados: shutdown 2024-03-25T18:26:56.509 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:26:58.973 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:11:01.011 34931== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:59.022 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:11:01.060 34931== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:59.055 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:11:01.093 34931== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:59.084 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:11:01.122 34935== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:59.092 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:11:01.135 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:59.092 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:11:01.130 34935== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:59.095 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:11:01.138 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:59.117 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:11:01.160 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:59.117 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:11:01.156 34935== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:03.315 DEBUG:teuthology.orchestra.run.smithi097:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:27:03.421 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.420+0000 7f5d43d3b640 10 monclient: get_monmap_and_config 2024-03-25T18:27:03.421 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.420+0000 7f5d43d3b640 10 monclient: build_initial_monmap 2024-03-25T18:27:03.421 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.420+0000 7f5d43d3b640 10 monclient: monmap: 2024-03-25T18:27:03.421 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:27:03.421 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:27:03.422 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:27:03.420986+0000 2024-03-25T18:27:03.422 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:27:03.420986+0000 2024-03-25T18:27:03.422 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:03.422 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:27:03.422 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:27:03.422 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T18:27:03.422 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-b 2024-03-25T18:27:03.422 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:27:03.424 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.421+0000 7f5d43d3b640 1 Processor -- start 2024-03-25T18:27:03.424 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.421+0000 7f5d43d3b640 1 -- start start 2024-03-25T18:27:03.424 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.421+0000 7f5d43d3b640 10 monclient: init 2024-03-25T18:27:03.424 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.422+0000 7f5d43d3b640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:03.424 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.422+0000 7f5d43d3b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:27:03.424 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.422+0000 7f5d43d3b640 10 monclient(hunting): picked mon.noname-b con 0x7f5d3c0b4d60 addr v1:172.21.15.97:6790/0 2024-03-25T18:27:03.424 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.422+0000 7f5d43d3b640 10 monclient(hunting): picked mon.noname-a con 0x7f5d3c0b1730 addr v1:172.21.15.97:6789/0 2024-03-25T18:27:03.424 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.422+0000 7f5d43d3b640 10 monclient(hunting): picked mon.noname-c con 0x7f5d3c14be90 addr v1:172.21.15.98:6789/0 2024-03-25T18:27:03.425 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.422+0000 7f5d43d3b640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d3c0af940 con 0x7f5d3c0b1730 2024-03-25T18:27:03.425 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.422+0000 7f5d43d3b640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d3c0b0a70 con 0x7f5d3c14be90 2024-03-25T18:27:03.425 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.422+0000 7f5d43d3b640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d3c0b0d50 con 0x7f5d3c0b4d60 2024-03-25T18:27:03.425 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.422+0000 7f5d43d3b640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:03.425 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.422+0000 7f5d43d3b640 10 monclient(hunting): authenticate will time out at 8235.697238s 2024-03-25T18:27:03.440 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.439+0000 7f5d41ad7640 1 --1- >> v1:172.21.15.97:6790/0 conn(0x7f5d3c0b4d60 0x7f5d3c13f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6790/0 says I am v1:172.21.15.97:49574/0 (socket says 172.21.15.97:49574) 2024-03-25T18:27:03.440 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.439+0000 7f5d41ad7640 1 -- 172.21.15.97:0/87345941 learned_addr learned my addr 172.21.15.97:0/87345941 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:27:03.451 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.450+0000 7f5d40ad5640 1 -- 172.21.15.97:0/87345941 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3466613360 0 0) 0x7f5d3c0b0d50 con 0x7f5d3c0b4d60 2024-03-25T18:27:03.451 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.450+0000 7f5d40ad5640 10 monclient(hunting): _negotiate 2024-03-25T18:27:03.451 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.450+0000 7f5d40ad5640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:03.451 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.450+0000 7f5d40ad5640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:03.451 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.450+0000 7f5d40ad5640 10 monclient(hunting): my global_id is 4688 2024-03-25T18:27:03.452 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.450+0000 7f5d40ad5640 1 -- 172.21.15.97:0/87345941 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d28003520 con 0x7f5d3c0b4d60 2024-03-25T18:27:03.453 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.452+0000 7f5d40ad5640 1 -- 172.21.15.97:0/87345941 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2701996878 0 0) 0x7f5d3c0b0a70 con 0x7f5d3c14be90 2024-03-25T18:27:03.453 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.452+0000 7f5d40ad5640 10 monclient(hunting): _negotiate 2024-03-25T18:27:03.453 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.452+0000 7f5d40ad5640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:03.453 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.452+0000 7f5d40ad5640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:03.453 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.452+0000 7f5d40ad5640 10 monclient(hunting): my global_id is 4714 2024-03-25T18:27:03.453 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.452+0000 7f5d40ad5640 1 -- 172.21.15.97:0/87345941 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d3c0b0d50 con 0x7f5d3c14be90 2024-03-25T18:27:03.454 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.454+0000 7f5d40ad5640 1 -- 172.21.15.97:0/87345941 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2243132810 0 0) 0x7f5d3c0af940 con 0x7f5d3c0b1730 2024-03-25T18:27:03.454 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.454+0000 7f5d40ad5640 10 monclient(hunting): _negotiate 2024-03-25T18:27:03.455 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.454+0000 7f5d40ad5640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:03.455 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.454+0000 7f5d40ad5640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:03.455 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.454+0000 7f5d40ad5640 10 monclient(hunting): my global_id is 4704 2024-03-25T18:27:03.455 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.454+0000 7f5d40ad5640 1 -- 172.21.15.97:0/87345941 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d3c0b0a70 con 0x7f5d3c0b1730 2024-03-25T18:27:03.469 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.468+0000 7f5d40ad5640 1 -- 172.21.15.97:0/87345941 <== mon.2 v1:172.21.15.97:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2672702517 0 0) 0x7f5d28003520 con 0x7f5d3c0b4d60 2024-03-25T18:27:03.469 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.468+0000 7f5d40ad5640 1 -- 172.21.15.97:0/87345941 >> v1:172.21.15.98:6789/0 conn(0x7f5d3c14be90 legacy=0x7f5d3c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:03.469 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.468+0000 7f5d40ad5640 1 -- 172.21.15.97:0/87345941 >> v1:172.21.15.97:6789/0 conn(0x7f5d3c0b1730 legacy=0x7f5d3c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:03.469 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.468+0000 7f5d40ad5640 10 monclient: _finish_hunting 0 2024-03-25T18:27:03.469 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.468+0000 7f5d40ad5640 1 monclient: found mon.noname-b 2024-03-25T18:27:03.469 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.468+0000 7f5d40ad5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:03.470 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.468+0000 7f5d40ad5640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.97:6790/0 2024-03-25T18:27:03.470 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.468+0000 7f5d40ad5640 1 -- 172.21.15.97:0/87345941 --> v1:172.21.15.97:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d3c0b1030 con 0x7f5d3c0b4d60 2024-03-25T18:27:03.470 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.468+0000 7f5d40ad5640 10 monclient: _finish_auth 0 2024-03-25T18:27:03.470 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.468+0000 7f5d40ad5640 10 monclient: _check_auth_tickets 2024-03-25T18:27:03.470 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.468+0000 7f5d40ad5640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:03.470 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.468+0000 7f5d40ad5640 1 -- 172.21.15.97:0/87345941 <== mon.2 v1:172.21.15.97:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f5d38001690 con 0x7f5d3c0b4d60 2024-03-25T18:27:03.470 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.468+0000 7f5d43d3b640 5 monclient: authenticate success, global_id 4688 2024-03-25T18:27:03.470 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.468+0000 7f5d43d3b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:27:03.470 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.468+0000 7f5d40ad5640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:03.470 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.468+0000 7f5d40ad5640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:27:03.470 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.468+0000 7f5d40ad5640 10 monclient: dump: 2024-03-25T18:27:03.471 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:27:03.471 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:27:03.471 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:27:03.471 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:27:03.471 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:03.471 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:27:03.471 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:27:03.471 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:27:03.471 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:27:03.471 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:27:03.471 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.468+0000 7f5d43d3b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:27:03.483 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.482+0000 7f5d40ad5640 1 -- 172.21.15.97:0/87345941 <== mon.2 v1:172.21.15.97:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5d380017f0 con 0x7f5d3c0b4d60 2024-03-25T18:27:03.483 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.482+0000 7f5d40ad5640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:03.483 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.482+0000 7f5d40ad5640 1 -- 172.21.15.97:0/87345941 <== mon.2 v1:172.21.15.97:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f5d38001c00 con 0x7f5d3c0b4d60 2024-03-25T18:27:03.483 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.482+0000 7f5d40ad5640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:03.483 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.482+0000 7f5d40ad5640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:27:03.483 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.482+0000 7f5d40ad5640 10 monclient: dump: 2024-03-25T18:27:03.484 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:27:03.484 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:27:03.484 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:27:03.484 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:27:03.484 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:03.484 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:27:03.484 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:27:03.484 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:27:03.484 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:27:03.484 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:27:03.484 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.482+0000 7f5d43d3b640 10 monclient: get_monmap_and_config success 2024-03-25T18:27:03.484 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.482+0000 7f5d43d3b640 4 set_mon_vals no callback set 2024-03-25T18:27:03.487 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.483+0000 7f5d43d3b640 10 monclient: shutdown 2024-03-25T18:27:03.487 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.483+0000 7f5d43d3b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:27:03.487 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.483+0000 7f5d43d3b640 1 -- 172.21.15.97:0/87345941 >> v1:172.21.15.97:6790/0 conn(0x7f5d3c0b4d60 legacy=0x7f5d3c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:03.487 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.483+0000 7f5d43d3b640 1 -- 172.21.15.97:0/87345941 shutdown_connections 2024-03-25T18:27:03.488 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.483+0000 7f5d43d3b640 1 -- 172.21.15.97:0/87345941 >> 172.21.15.97:0/87345941 conn(0x7f5d3c0617a0 msgr2=0x7f5d3c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:03.488 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.483+0000 7f5d43d3b640 1 -- 172.21.15.97:0/87345941 shutdown_connections 2024-03-25T18:27:03.488 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.484+0000 7f5d43d3b640 1 -- 172.21.15.97:0/87345941 wait complete. 2024-03-25T18:27:03.488 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.484+0000 7f5d43d3b640 10 monclient: build_initial_monmap 2024-03-25T18:27:03.488 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.484+0000 7f5d43d3b640 10 monclient: monmap: 2024-03-25T18:27:03.488 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:27:03.488 INFO:teuthology.misc.health.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:27:03.488 INFO:teuthology.misc.health.smithi097.stderr:last_changed 0.000000 2024-03-25T18:27:03.488 INFO:teuthology.misc.health.smithi097.stderr:created 0.000000 2024-03-25T18:27:03.488 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:03.488 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 1 2024-03-25T18:27:03.488 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:27:03.489 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T18:27:03.489 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-c 2024-03-25T18:27:03.489 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:27:03.489 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.484+0000 7f5d43d3b640 1 librados: starting msgr at 2024-03-25T18:27:03.489 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.484+0000 7f5d43d3b640 1 librados: starting objecter 2024-03-25T18:27:03.489 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.485+0000 7f5d43d3b640 1 Processor -- start 2024-03-25T18:27:03.489 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.485+0000 7f5d43d3b640 1 -- start start 2024-03-25T18:27:03.489 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.485+0000 7f5d43d3b640 1 librados: setting wanted keys 2024-03-25T18:27:03.489 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.485+0000 7f5d43d3b640 1 librados: calling monclient init 2024-03-25T18:27:03.489 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.485+0000 7f5d43d3b640 10 monclient: init 2024-03-25T18:27:03.489 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.485+0000 7f5d43d3b640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:03.489 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.485+0000 7f5d43d3b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:27:03.489 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.485+0000 7f5d43d3b640 10 monclient(hunting): picked mon.noname-c con 0x7f5d3c0b1730 addr v1:172.21.15.97:6790/0 2024-03-25T18:27:03.490 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.485+0000 7f5d43d3b640 10 monclient(hunting): picked mon.noname-a con 0x7f5d3c0b4d60 addr v1:172.21.15.97:6789/0 2024-03-25T18:27:03.490 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.485+0000 7f5d43d3b640 10 monclient(hunting): picked mon.noname-b con 0x7f5d3c14be90 addr v1:172.21.15.98:6789/0 2024-03-25T18:27:03.490 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.485+0000 7f5d43d3b640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d3c1db350 con 0x7f5d3c0b4d60 2024-03-25T18:27:03.490 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.485+0000 7f5d43d3b640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d3c1378b0 con 0x7f5d3c14be90 2024-03-25T18:27:03.490 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.485+0000 7f5d43d3b640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d3c137b90 con 0x7f5d3c0b1730 2024-03-25T18:27:03.490 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.485+0000 7f5d43d3b640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:03.490 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.485+0000 7f5d43d3b640 10 monclient(hunting): authenticate will time out at 8235.760707s 2024-03-25T18:27:03.494 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.494+0000 7f5d41ad7640 1 --1- >> v1:172.21.15.97:6790/0 conn(0x7f5d3c0b1730 0x7f5d3c1db7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.97:6790/0 says I am v1:172.21.15.97:49610/0 (socket says 172.21.15.97:49610) 2024-03-25T18:27:03.494 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.494+0000 7f5d41ad7640 1 -- 172.21.15.97:0/1822862725 learned_addr learned my addr 172.21.15.97:0/1822862725 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:27:03.496 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.496+0000 7f5d327fc640 10 client.?.objecter ms_handle_connect 0x7f5d3c0b1730 2024-03-25T18:27:03.496 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.496+0000 7f5d327fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:03.500 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.499+0000 7f5d327fc640 10 client.?.objecter ms_handle_connect 0x7f5d3c14be90 2024-03-25T18:27:03.500 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.499+0000 7f5d327fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:03.506 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.506+0000 7f5d327fc640 1 -- 172.21.15.97:0/1822862725 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3651225092 0 0) 0x7f5d3c1378b0 con 0x7f5d3c14be90 2024-03-25T18:27:03.507 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.506+0000 7f5d327fc640 10 monclient(hunting): _negotiate 2024-03-25T18:27:03.507 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.506+0000 7f5d327fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:03.507 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.506+0000 7f5d327fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:03.507 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.506+0000 7f5d327fc640 10 monclient(hunting): my global_id is 4717 2024-03-25T18:27:03.507 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.506+0000 7f5d327fc640 1 -- 172.21.15.97:0/1822862725 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d20004eb0 con 0x7f5d3c14be90 2024-03-25T18:27:03.510 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.509+0000 7f5d327fc640 1 -- 172.21.15.97:0/1822862725 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1367685367 0 0) 0x7f5d3c137b90 con 0x7f5d3c0b1730 2024-03-25T18:27:03.510 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.509+0000 7f5d327fc640 10 monclient(hunting): _negotiate 2024-03-25T18:27:03.510 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.509+0000 7f5d327fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:03.510 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.509+0000 7f5d327fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:03.510 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.509+0000 7f5d327fc640 10 monclient(hunting): my global_id is 4691 2024-03-25T18:27:03.510 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.509+0000 7f5d327fc640 1 -- 172.21.15.97:0/1822862725 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d3c1378b0 con 0x7f5d3c0b1730 2024-03-25T18:27:03.511 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.510+0000 7f5d327fc640 10 client.?.objecter ms_handle_connect 0x7f5d3c0b4d60 2024-03-25T18:27:03.511 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.510+0000 7f5d327fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:03.525 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.524+0000 7f5d327fc640 1 -- 172.21.15.97:0/1822862725 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 599998915 0 0) 0x7f5d20004eb0 con 0x7f5d3c14be90 2024-03-25T18:27:03.525 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.524+0000 7f5d327fc640 1 -- 172.21.15.97:0/1822862725 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5d3c137b90 con 0x7f5d3c14be90 2024-03-25T18:27:03.525 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.524+0000 7f5d327fc640 1 -- 172.21.15.97:0/1822862725 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f5d34001bd0 con 0x7f5d3c14be90 2024-03-25T18:27:03.525 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.524+0000 7f5d327fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:03.525 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.524+0000 7f5d327fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:27:03.525 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.524+0000 7f5d327fc640 10 monclient(hunting): dump: 2024-03-25T18:27:03.525 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:27:03.525 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:27:03.526 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:27:03.526 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:27:03.526 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:03.526 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:27:03.526 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:27:03.526 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:27:03.526 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:27:03.526 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:27:03.526 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.524+0000 7f5d327fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:27:03.536 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.535+0000 7f5d327fc640 1 -- 172.21.15.97:0/1822862725 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2870202449 0 0) 0x7f5d3c1db350 con 0x7f5d3c0b4d60 2024-03-25T18:27:03.536 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.535+0000 7f5d327fc640 10 monclient(hunting): _negotiate 2024-03-25T18:27:03.536 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.535+0000 7f5d327fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:03.536 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.535+0000 7f5d327fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:03.536 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.535+0000 7f5d327fc640 10 monclient(hunting): my global_id is 4707 2024-03-25T18:27:03.536 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.535+0000 7f5d327fc640 1 -- 172.21.15.97:0/1822862725 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d20004eb0 con 0x7f5d3c0b4d60 2024-03-25T18:27:03.547 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.546+0000 7f5d327fc640 1 -- 172.21.15.97:0/1822862725 <== mon.2 v1:172.21.15.97:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1359584261 0 0) 0x7f5d3c1378b0 con 0x7f5d3c0b1730 2024-03-25T18:27:03.547 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.546+0000 7f5d327fc640 1 -- 172.21.15.97:0/1822862725 --> v1:172.21.15.97:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5d3c1db350 con 0x7f5d3c0b1730 2024-03-25T18:27:03.547 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.546+0000 7f5d327fc640 1 -- 172.21.15.97:0/1822862725 <== mon.2 v1:172.21.15.97:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f5d38001b80 con 0x7f5d3c0b1730 2024-03-25T18:27:03.547 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.546+0000 7f5d327fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:03.547 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.546+0000 7f5d327fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:27:03.547 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.546+0000 7f5d327fc640 10 monclient(hunting): dump: 2024-03-25T18:27:03.547 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:27:03.547 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:27:03.547 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:27:03.547 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:27:03.548 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:03.548 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:27:03.548 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:27:03.548 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:27:03.548 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:27:03.548 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:27:03.551 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.549+0000 7f5d327fc640 1 -- 172.21.15.97:0/1822862725 <== mon.1 v1:172.21.15.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2065472082 0 0) 0x7f5d3c137b90 con 0x7f5d3c14be90 2024-03-25T18:27:03.551 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.549+0000 7f5d327fc640 1 -- 172.21.15.97:0/1822862725 >> v1:172.21.15.97:6790/0 conn(0x7f5d3c0b1730 legacy=0x7f5d3c1db7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:03.551 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.549+0000 7f5d327fc640 1 -- 172.21.15.97:0/1822862725 >> v1:172.21.15.97:6789/0 conn(0x7f5d3c0b4d60 legacy=0x7f5d3c141a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:03.551 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.549+0000 7f5d327fc640 10 monclient: _finish_hunting 0 2024-03-25T18:27:03.551 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.549+0000 7f5d327fc640 1 monclient: found mon.b 2024-03-25T18:27:03.551 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.549+0000 7f5d327fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:03.551 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.549+0000 7f5d327fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:27:03.551 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.549+0000 7f5d327fc640 1 -- 172.21.15.97:0/1822862725 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d3c137e70 con 0x7f5d3c14be90 2024-03-25T18:27:03.551 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.549+0000 7f5d327fc640 10 monclient: _finish_auth 0 2024-03-25T18:27:03.551 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.549+0000 7f5d327fc640 10 monclient: _check_auth_tickets 2024-03-25T18:27:03.551 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.549+0000 7f5d327fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:03.551 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.549+0000 7f5d43d3b640 5 monclient: authenticate success, global_id 4717 2024-03-25T18:27:03.552 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.549+0000 7f5d43d3b640 10 monclient: _renew_subs 2024-03-25T18:27:03.552 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.549+0000 7f5d43d3b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:27:03.552 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.549+0000 7f5d43d3b640 1 -- 172.21.15.97:0/1822862725 --> v1:172.21.15.98:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d3c1dbef0 con 0x7f5d3c14be90 2024-03-25T18:27:03.552 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.549+0000 7f5d43d3b640 10 client.4717.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:27:03.552 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.549+0000 7f5d43d3b640 10 monclient: _renew_subs 2024-03-25T18:27:03.552 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.549+0000 7f5d43d3b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:27:03.552 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.549+0000 7f5d43d3b640 1 -- 172.21.15.97:0/1822862725 --> v1:172.21.15.98:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d3c137010 con 0x7f5d3c14be90 2024-03-25T18:27:03.552 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.549+0000 7f5d43d3b640 1 librados: init done 2024-03-25T18:27:03.553 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.551+0000 7f5d43d3b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:03.553 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.551+0000 7f5d43d3b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:03.553 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.551+0000 7f5d43d3b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:27:03.553 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.551+0000 7f5d43d3b640 1 -- 172.21.15.97:0/1822862725 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d3c1dc0e0 con 0x7f5d3c14be90 2024-03-25T18:27:03.839 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.838+0000 7f5d327fc640 1 -- 172.21.15.97:0/1822862725 <== mon.1 v1:172.21.15.98:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5d34003970 con 0x7f5d3c14be90 2024-03-25T18:27:03.839 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.838+0000 7f5d327fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:03.839 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.838+0000 7f5d327fc640 1 -- 172.21.15.97:0/1822862725 <== mon.1 v1:172.21.15.98:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f5d34002100 con 0x7f5d3c14be90 2024-03-25T18:27:03.839 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.838+0000 7f5d327fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:03.839 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.838+0000 7f5d327fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:27:03.839 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.838+0000 7f5d327fc640 10 monclient: dump: 2024-03-25T18:27:03.839 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:27:03.840 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:27:03.840 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:27:03.840 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:27:03.840 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:03.840 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:27:03.840 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:27:03.840 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:27:03.840 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:27:03.840 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:27:03.840 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.838+0000 7f5d33fff640 4 set_mon_vals no callback set 2024-03-25T18:27:03.841 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.839+0000 7f5d327fc640 1 -- 172.21.15.97:0/1822862725 <== mon.1 v1:172.21.15.98:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2377622137 0 0) 0x7f5d34019950 con 0x7f5d3c14be90 2024-03-25T18:27:03.841 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.840+0000 7f5d41ad7640 10 monclient: get_auth_request con 0x7f5d2006df70 auth_method 0 2024-03-25T18:27:03.841 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.840+0000 7f5d327fc640 10 client.4717.objecter ms_handle_connect 0x7f5d2006df70 2024-03-25T18:27:03.846 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.845+0000 7f5d327fc640 1 -- 172.21.15.97:0/1822862725 <== mon.1 v1:172.21.15.98:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3078631805 0 0) 0x7f5d34053600 con 0x7f5d3c14be90 2024-03-25T18:27:03.846 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.845+0000 7f5d327fc640 10 client.4717.objecter ms_dispatch 0x7f5d3c000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:27:03.846 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.845+0000 7f5d327fc640 3 client.4717.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:27:03.846 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.845+0000 7f5d327fc640 3 client.4717.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:27:03.846 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.845+0000 7f5d327fc640 20 client.4717.objecter dump_active .. 0 homeless 2024-03-25T18:27:03.846 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.845+0000 7f5d327fc640 1 -- 172.21.15.97:0/1822862725 <== mon.1 v1:172.21.15.98:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5d3404ef40 con 0x7f5d3c14be90 2024-03-25T18:27:03.846 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.845+0000 7f5d327fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:03.846 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:03.845+0000 7f5d327fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:27:04.098 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:04.097+0000 7f5d43d3b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:04.098 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:04.097+0000 7f5d43d3b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:04.098 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:04.097+0000 7f5d43d3b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:27:04.099 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:04.097+0000 7f5d43d3b640 1 -- 172.21.15.97:0/1822862725 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5d3c1db500 con 0x7f5d3c14be90 2024-03-25T18:27:04.131 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:04.130+0000 7f5d327fc640 1 -- 172.21.15.97:0/1822862725 <== mon.1 v1:172.21.15.98:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 337131954) 0x7f5d3c1db500 con 0x7f5d3c14be90 2024-03-25T18:27:04.131 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:04.130+0000 7f5d327fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:04.131 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:04.130+0000 7f5d327fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:27:04.132 INFO:teuthology.misc.health.smithi097.stdout: 2024-03-25T18:27:04.132 INFO:teuthology.misc.health.smithi097.stdout:{"epoch":11,"fsid":"4c3d472c-79b8-486b-9e3d-2738d4306754","created":"2024-03-25T18:16:14.737309+0000","modified":"2024-03-25T18:26:18.750062+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:18.750062+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cb5152b9-262f-45c3-94b8-18d0e81df558","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"6daf8793-0838-473d-a7d6-9a0d2894c75d","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"136f1493-3b9c-4f43-a2e7-641482d69dae","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"397d538c-4f64-4daf-9e8b-662d09f91327","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4f1df08-663f-4bda-93f8-3878ab7c24b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c4b18d70-a3ce-4b15-9b70-eae5525bd761","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8dd4033a-de0e-4f8a-96dc-59f3c1f825b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0077722e-ffa6-4a3d-9891-929d9869b286","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:27:04.134 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:04.132+0000 7f5d43d3b640 10 librados: watch_flush enter 2024-03-25T18:27:04.134 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:04.133+0000 7f5d43d3b640 10 librados: watch_flush exit 2024-03-25T18:27:04.134 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:04.133+0000 7f5d43d3b640 20 client.4717.objecter shutdown clearing up homeless session... 2024-03-25T18:27:04.134 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:04.133+0000 7f5d43d3b640 10 client.4717.objecter successfully canceled tick 2024-03-25T18:27:04.134 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:04.133+0000 7f5d43d3b640 1 -- 172.21.15.97:0/1822862725 >> v1:172.21.15.98:6800/800387573 conn(0x7f5d2006df70 legacy=0x7f5d20070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:04.134 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:04.133+0000 7f5d43d3b640 10 monclient: shutdown 2024-03-25T18:27:04.134 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:04.133+0000 7f5d43d3b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:27:04.134 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:04.133+0000 7f5d43d3b640 1 -- 172.21.15.97:0/1822862725 >> v1:172.21.15.98:6789/0 conn(0x7f5d3c14be90 legacy=0x7f5d3c1e34d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:04.135 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:04.133+0000 7f5d43d3b640 1 -- 172.21.15.97:0/1822862725 shutdown_connections 2024-03-25T18:27:04.135 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:04.133+0000 7f5d43d3b640 1 -- 172.21.15.97:0/1822862725 >> 172.21.15.97:0/1822862725 conn(0x7f5d3c0617a0 msgr2=0x7f5d3c0b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:04.135 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:04.133+0000 7f5d43d3b640 1 -- 172.21.15.97:0/1822862725 shutdown_connections 2024-03-25T18:27:04.135 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:04.133+0000 7f5d43d3b640 1 -- 172.21.15.97:0/1822862725 wait complete. 2024-03-25T18:27:04.135 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:04.133+0000 7f5d43d3b640 1 librados: shutdown 2024-03-25T18:27:04.145 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:27:10.952 DEBUG:teuthology.orchestra.run.smithi097:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:27:11.065 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.064+0000 7fbf33251640 10 monclient: get_monmap_and_config 2024-03-25T18:27:11.065 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.064+0000 7fbf33251640 10 monclient: build_initial_monmap 2024-03-25T18:27:11.065 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.064+0000 7fbf33251640 10 monclient: monmap: 2024-03-25T18:27:11.065 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:27:11.065 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:27:11.065 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:27:11.064914+0000 2024-03-25T18:27:11.065 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:27:11.064914+0000 2024-03-25T18:27:11.065 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:11.066 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:27:11.066 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:27:11.066 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T18:27:11.066 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-b 2024-03-25T18:27:11.066 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:27:11.067 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.065+0000 7fbf33251640 1 Processor -- start 2024-03-25T18:27:11.067 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.065+0000 7fbf33251640 1 -- start start 2024-03-25T18:27:11.068 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.065+0000 7fbf33251640 10 monclient: init 2024-03-25T18:27:11.068 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.066+0000 7fbf33251640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:11.068 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.066+0000 7fbf33251640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:27:11.068 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.066+0000 7fbf33251640 10 monclient(hunting): picked mon.noname-a con 0x7fbf2c0b4d60 addr v1:172.21.15.97:6789/0 2024-03-25T18:27:11.068 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.066+0000 7fbf33251640 10 monclient(hunting): picked mon.noname-b con 0x7fbf2c0b1730 addr v1:172.21.15.97:6790/0 2024-03-25T18:27:11.068 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.066+0000 7fbf33251640 10 monclient(hunting): picked mon.noname-c con 0x7fbf2c14be90 addr v1:172.21.15.98:6789/0 2024-03-25T18:27:11.068 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.066+0000 7fbf33251640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf2c0af940 con 0x7fbf2c0b4d60 2024-03-25T18:27:11.068 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.066+0000 7fbf33251640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf2c0b0a70 con 0x7fbf2c14be90 2024-03-25T18:27:11.068 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.066+0000 7fbf33251640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf2c0b0d50 con 0x7fbf2c0b1730 2024-03-25T18:27:11.068 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.066+0000 7fbf33251640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:11.068 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.066+0000 7fbf33251640 10 monclient(hunting): authenticate will time out at 8243.341644s 2024-03-25T18:27:11.076 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.076+0000 7fbf317ee640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fbf2c14be90 0x7fbf2c14e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.97:37826/0 (socket says 172.21.15.97:37826) 2024-03-25T18:27:11.076 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.076+0000 7fbf317ee640 1 -- 172.21.15.97:0/1617355944 learned_addr learned my addr 172.21.15.97:0/1617355944 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:27:11.099 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.098+0000 7fbf2b7fe640 1 -- 172.21.15.97:0/1617355944 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4003005081 0 0) 0x7fbf2c0af940 con 0x7fbf2c0b4d60 2024-03-25T18:27:11.099 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.098+0000 7fbf2b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:27:11.099 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.098+0000 7fbf2b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:11.099 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.098+0000 7fbf2b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:11.099 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.098+0000 7fbf2b7fe640 10 monclient(hunting): my global_id is 4710 2024-03-25T18:27:11.099 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.098+0000 7fbf2b7fe640 1 -- 172.21.15.97:0/1617355944 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf1c003520 con 0x7fbf2c0b4d60 2024-03-25T18:27:11.156 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.155+0000 7fbf2b7fe640 1 -- 172.21.15.97:0/1617355944 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3347089941 0 0) 0x7fbf1c003520 con 0x7fbf2c0b4d60 2024-03-25T18:27:11.157 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.155+0000 7fbf2b7fe640 1 -- 172.21.15.97:0/1617355944 >> v1:172.21.15.97:6790/0 conn(0x7fbf2c0b1730 legacy=0x7fbf2c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:11.157 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.155+0000 7fbf2b7fe640 1 -- 172.21.15.97:0/1617355944 >> v1:172.21.15.98:6789/0 conn(0x7fbf2c14be90 legacy=0x7fbf2c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:11.157 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.155+0000 7fbf2b7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:27:11.157 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.155+0000 7fbf2b7fe640 1 monclient: found mon.noname-a 2024-03-25T18:27:11.157 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.155+0000 7fbf2b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:11.157 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.155+0000 7fbf2b7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.97:6789/0 2024-03-25T18:27:11.157 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.155+0000 7fbf2b7fe640 1 -- 172.21.15.97:0/1617355944 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf2c0b1030 con 0x7fbf2c0b4d60 2024-03-25T18:27:11.157 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.155+0000 7fbf2b7fe640 10 monclient: _finish_auth 0 2024-03-25T18:27:11.157 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.155+0000 7fbf2b7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:27:11.157 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.155+0000 7fbf2b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:11.157 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.155+0000 7fbf2b7fe640 1 -- 172.21.15.97:0/1617355944 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7fbf20001690 con 0x7fbf2c0b4d60 2024-03-25T18:27:11.157 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.155+0000 7fbf33251640 5 monclient: authenticate success, global_id 4710 2024-03-25T18:27:11.157 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.155+0000 7fbf33251640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:27:11.157 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.155+0000 7fbf2b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:11.158 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.155+0000 7fbf2b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:27:11.158 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.155+0000 7fbf2b7fe640 10 monclient: dump: 2024-03-25T18:27:11.158 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:27:11.158 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:27:11.158 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:27:11.158 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:27:11.158 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:11.158 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:27:11.158 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:27:11.158 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:27:11.158 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:27:11.158 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:27:11.158 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.155+0000 7fbf33251640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:27:11.175 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.174+0000 7fbf2b7fe640 1 -- 172.21.15.97:0/1617355944 <== mon.0 v1:172.21.15.97:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbf200017f0 con 0x7fbf2c0b4d60 2024-03-25T18:27:11.175 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.174+0000 7fbf2b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:11.175 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.174+0000 7fbf2b7fe640 1 -- 172.21.15.97:0/1617355944 <== mon.0 v1:172.21.15.97:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7fbf20001c00 con 0x7fbf2c0b4d60 2024-03-25T18:27:11.175 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.174+0000 7fbf33251640 10 monclient: get_monmap_and_config success 2024-03-25T18:27:11.175 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.174+0000 7fbf33251640 4 set_mon_vals no callback set 2024-03-25T18:27:11.175 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.174+0000 7fbf2b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:11.175 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.174+0000 7fbf2b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:27:11.175 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.174+0000 7fbf2b7fe640 10 monclient: dump: 2024-03-25T18:27:11.175 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:27:11.175 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:27:11.175 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:27:11.175 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:27:11.176 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:11.176 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:27:11.176 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:27:11.176 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:27:11.176 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:27:11.176 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:27:11.177 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.175+0000 7fbf33251640 10 monclient: shutdown 2024-03-25T18:27:11.177 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.175+0000 7fbf33251640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:27:11.177 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.175+0000 7fbf33251640 1 -- 172.21.15.97:0/1617355944 >> v1:172.21.15.97:6789/0 conn(0x7fbf2c0b4d60 legacy=0x7fbf2c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:11.177 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.175+0000 7fbf33251640 1 -- 172.21.15.97:0/1617355944 shutdown_connections 2024-03-25T18:27:11.177 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.175+0000 7fbf33251640 1 -- 172.21.15.97:0/1617355944 >> 172.21.15.97:0/1617355944 conn(0x7fbf2c0617a0 msgr2=0x7fbf2c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:11.177 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.175+0000 7fbf33251640 1 -- 172.21.15.97:0/1617355944 shutdown_connections 2024-03-25T18:27:11.178 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.175+0000 7fbf33251640 1 -- 172.21.15.97:0/1617355944 wait complete. 2024-03-25T18:27:11.178 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.176+0000 7fbf33251640 10 monclient: build_initial_monmap 2024-03-25T18:27:11.178 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.176+0000 7fbf33251640 10 monclient: monmap: 2024-03-25T18:27:11.178 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:27:11.178 INFO:teuthology.misc.health.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:27:11.178 INFO:teuthology.misc.health.smithi097.stderr:last_changed 0.000000 2024-03-25T18:27:11.178 INFO:teuthology.misc.health.smithi097.stderr:created 0.000000 2024-03-25T18:27:11.178 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:11.178 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 1 2024-03-25T18:27:11.178 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:27:11.178 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T18:27:11.178 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-c 2024-03-25T18:27:11.178 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:27:11.179 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.176+0000 7fbf33251640 1 librados: starting msgr at 2024-03-25T18:27:11.179 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.176+0000 7fbf33251640 1 librados: starting objecter 2024-03-25T18:27:11.179 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.176+0000 7fbf33251640 1 Processor -- start 2024-03-25T18:27:11.179 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.176+0000 7fbf33251640 1 -- start start 2024-03-25T18:27:11.179 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.176+0000 7fbf33251640 1 librados: setting wanted keys 2024-03-25T18:27:11.179 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.176+0000 7fbf33251640 1 librados: calling monclient init 2024-03-25T18:27:11.179 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.176+0000 7fbf33251640 10 monclient: init 2024-03-25T18:27:11.180 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.177+0000 7fbf33251640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:11.180 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.177+0000 7fbf33251640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:27:11.180 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.177+0000 7fbf33251640 10 monclient(hunting): picked mon.noname-b con 0x7fbf2c0b1730 addr v1:172.21.15.98:6789/0 2024-03-25T18:27:11.180 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.177+0000 7fbf33251640 10 monclient(hunting): picked mon.noname-c con 0x7fbf2c0b4d60 addr v1:172.21.15.97:6790/0 2024-03-25T18:27:11.180 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.177+0000 7fbf33251640 10 monclient(hunting): picked mon.noname-a con 0x7fbf2c14be90 addr v1:172.21.15.97:6789/0 2024-03-25T18:27:11.180 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.177+0000 7fbf33251640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf2c13bde0 con 0x7fbf2c14be90 2024-03-25T18:27:11.180 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.177+0000 7fbf33251640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf2c140f50 con 0x7fbf2c0b1730 2024-03-25T18:27:11.180 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.177+0000 7fbf33251640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf2c1410e0 con 0x7fbf2c0b4d60 2024-03-25T18:27:11.180 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.177+0000 7fbf33251640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:11.180 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.177+0000 7fbf33251640 10 monclient(hunting): authenticate will time out at 8243.452778s 2024-03-25T18:27:11.187 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.186+0000 7fbf317ee640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7fbf2c14be90 0x7fbf2c142bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:56374/0 (socket says 172.21.15.97:56374) 2024-03-25T18:27:11.187 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.186+0000 7fbf317ee640 1 -- 172.21.15.97:0/3991127513 learned_addr learned my addr 172.21.15.97:0/3991127513 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:27:11.191 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.191+0000 7fbf297fa640 10 client.?.objecter ms_handle_connect 0x7fbf2c14be90 2024-03-25T18:27:11.191 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.191+0000 7fbf297fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:11.200 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.200+0000 7fbf297fa640 1 -- 172.21.15.97:0/3991127513 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1992139273 0 0) 0x7fbf2c13bde0 con 0x7fbf2c14be90 2024-03-25T18:27:11.200 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.200+0000 7fbf297fa640 10 monclient(hunting): _negotiate 2024-03-25T18:27:11.200 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.200+0000 7fbf297fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:11.200 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.200+0000 7fbf297fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:11.200 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.200+0000 7fbf297fa640 10 monclient(hunting): my global_id is 4713 2024-03-25T18:27:11.201 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.200+0000 7fbf297fa640 1 -- 172.21.15.97:0/3991127513 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf10004eb0 con 0x7fbf2c14be90 2024-03-25T18:27:11.214 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.213+0000 7fbf297fa640 1 -- 172.21.15.97:0/3991127513 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 462164690 0 0) 0x7fbf10004eb0 con 0x7fbf2c14be90 2024-03-25T18:27:11.214 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.213+0000 7fbf297fa640 1 -- 172.21.15.97:0/3991127513 --> v1:172.21.15.97:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbf2c13bde0 con 0x7fbf2c14be90 2024-03-25T18:27:11.214 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.213+0000 7fbf297fa640 1 -- 172.21.15.97:0/3991127513 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7fbf24001b10 con 0x7fbf2c14be90 2024-03-25T18:27:11.214 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.213+0000 7fbf297fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:11.214 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.213+0000 7fbf297fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:27:11.215 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.213+0000 7fbf297fa640 10 monclient(hunting): dump: 2024-03-25T18:27:11.215 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:27:11.215 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:27:11.215 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:27:11.215 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:27:11.215 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:11.215 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:27:11.215 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:27:11.215 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:27:11.215 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:27:11.215 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:27:11.215 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.213+0000 7fbf297fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:27:11.227 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.226+0000 7fbf297fa640 10 client.?.objecter ms_handle_connect 0x7fbf2c0b4d60 2024-03-25T18:27:11.227 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.226+0000 7fbf297fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:11.227 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.226+0000 7fbf297fa640 10 client.?.objecter ms_handle_connect 0x7fbf2c0b1730 2024-03-25T18:27:11.227 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.226+0000 7fbf297fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:11.233 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.233+0000 7fbf297fa640 1 -- 172.21.15.97:0/3991127513 <== mon.0 v1:172.21.15.97:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4267306547 0 0) 0x7fbf2c13bde0 con 0x7fbf2c14be90 2024-03-25T18:27:11.234 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.233+0000 7fbf297fa640 1 -- 172.21.15.97:0/3991127513 >> v1:172.21.15.97:6790/0 conn(0x7fbf2c0b4d60 legacy=0x7fbf2c1d7d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:11.234 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.233+0000 7fbf297fa640 1 -- 172.21.15.97:0/3991127513 >> v1:172.21.15.98:6789/0 conn(0x7fbf2c0b1730 legacy=0x7fbf2c140840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:11.234 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.233+0000 7fbf297fa640 10 monclient: _finish_hunting 0 2024-03-25T18:27:11.234 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.233+0000 7fbf297fa640 1 monclient: found mon.a 2024-03-25T18:27:11.234 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.233+0000 7fbf297fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:11.234 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.233+0000 7fbf297fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:27:11.234 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.233+0000 7fbf297fa640 1 -- 172.21.15.97:0/3991127513 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf2c1d8450 con 0x7fbf2c14be90 2024-03-25T18:27:11.234 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.233+0000 7fbf297fa640 10 monclient: _finish_auth 0 2024-03-25T18:27:11.234 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.233+0000 7fbf297fa640 10 monclient: _check_auth_tickets 2024-03-25T18:27:11.234 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.233+0000 7fbf297fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:11.234 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.233+0000 7fbf33251640 5 monclient: authenticate success, global_id 4713 2024-03-25T18:27:11.234 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.233+0000 7fbf33251640 10 monclient: _renew_subs 2024-03-25T18:27:11.234 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.233+0000 7fbf33251640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:27:11.235 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.233+0000 7fbf33251640 1 -- 172.21.15.97:0/3991127513 --> v1:172.21.15.97:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf2c1d86a0 con 0x7fbf2c14be90 2024-03-25T18:27:11.235 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.233+0000 7fbf33251640 10 client.4713.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:27:11.235 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.233+0000 7fbf33251640 10 monclient: _renew_subs 2024-03-25T18:27:11.235 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.233+0000 7fbf33251640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:27:11.235 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.233+0000 7fbf33251640 1 -- 172.21.15.97:0/3991127513 --> v1:172.21.15.97:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf2c1426d0 con 0x7fbf2c14be90 2024-03-25T18:27:11.235 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.233+0000 7fbf33251640 1 librados: init done 2024-03-25T18:27:11.235 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.234+0000 7fbf33251640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:11.236 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.234+0000 7fbf33251640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:11.236 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.234+0000 7fbf33251640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:27:11.236 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.234+0000 7fbf33251640 1 -- 172.21.15.97:0/3991127513 --> v1:172.21.15.97:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf2c1401f0 con 0x7fbf2c14be90 2024-03-25T18:27:11.543 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.542+0000 7fbf297fa640 1 -- 172.21.15.97:0/3991127513 <== mon.0 v1:172.21.15.97:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbf24001c70 con 0x7fbf2c14be90 2024-03-25T18:27:11.544 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.542+0000 7fbf297fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:11.544 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.542+0000 7fbf297fa640 1 -- 172.21.15.97:0/3991127513 <== mon.0 v1:172.21.15.97:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7fbf24004d00 con 0x7fbf2c14be90 2024-03-25T18:27:11.544 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.542+0000 7fbf297fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:11.544 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.542+0000 7fbf297fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:27:11.544 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.542+0000 7fbf297fa640 10 monclient: dump: 2024-03-25T18:27:11.544 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:27:11.544 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:27:11.544 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:27:11.544 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:27:11.544 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:11.544 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:27:11.544 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:27:11.544 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:27:11.545 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:27:11.545 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:27:11.545 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.542+0000 7fbf2affd640 4 set_mon_vals no callback set 2024-03-25T18:27:11.545 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.543+0000 7fbf297fa640 1 -- 172.21.15.97:0/3991127513 <== mon.0 v1:172.21.15.97:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2377622137 0 0) 0x7fbf24004f20 con 0x7fbf2c14be90 2024-03-25T18:27:11.545 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.544+0000 7fbf30fed640 10 monclient: get_auth_request con 0x7fbf1006dc20 auth_method 0 2024-03-25T18:27:11.545 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.544+0000 7fbf297fa640 10 client.4713.objecter ms_handle_connect 0x7fbf1006dc20 2024-03-25T18:27:11.555 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.554+0000 7fbf297fa640 1 -- 172.21.15.97:0/3991127513 <== mon.0 v1:172.21.15.97:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3078631805 0 0) 0x7fbf24054ab0 con 0x7fbf2c14be90 2024-03-25T18:27:11.555 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.554+0000 7fbf297fa640 10 client.4713.objecter ms_dispatch 0x7fbf2c000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:27:11.555 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.554+0000 7fbf297fa640 3 client.4713.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:27:11.555 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.554+0000 7fbf297fa640 3 client.4713.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:27:11.555 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.554+0000 7fbf297fa640 20 client.4713.objecter dump_active .. 0 homeless 2024-03-25T18:27:11.555 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.554+0000 7fbf297fa640 1 -- 172.21.15.97:0/3991127513 <== mon.0 v1:172.21.15.97:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbf240505c0 con 0x7fbf2c14be90 2024-03-25T18:27:11.555 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.554+0000 7fbf297fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:11.555 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.554+0000 7fbf297fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:27:11.801 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.800+0000 7fbf33251640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:11.801 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.800+0000 7fbf33251640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:11.801 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.800+0000 7fbf33251640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:27:11.801 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.800+0000 7fbf33251640 1 -- 172.21.15.97:0/3991127513 --> v1:172.21.15.97:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbf2c1d7a60 con 0x7fbf2c14be90 2024-03-25T18:27:11.841 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.840+0000 7fbf297fa640 1 -- 172.21.15.97:0/3991127513 <== mon.0 v1:172.21.15.97:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 337131954) 0x7fbf2c1d7a60 con 0x7fbf2c14be90 2024-03-25T18:27:11.842 INFO:teuthology.misc.health.smithi097.stdout: 2024-03-25T18:27:11.842 INFO:teuthology.misc.health.smithi097.stdout:{"epoch":11,"fsid":"4c3d472c-79b8-486b-9e3d-2738d4306754","created":"2024-03-25T18:16:14.737309+0000","modified":"2024-03-25T18:26:18.750062+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:18.750062+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cb5152b9-262f-45c3-94b8-18d0e81df558","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"6daf8793-0838-473d-a7d6-9a0d2894c75d","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"136f1493-3b9c-4f43-a2e7-641482d69dae","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"397d538c-4f64-4daf-9e8b-662d09f91327","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4f1df08-663f-4bda-93f8-3878ab7c24b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c4b18d70-a3ce-4b15-9b70-eae5525bd761","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8dd4033a-de0e-4f8a-96dc-59f3c1f825b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0077722e-ffa6-4a3d-9891-929d9869b286","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:27:11.842 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.840+0000 7fbf297fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:11.842 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.840+0000 7fbf297fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:27:11.844 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.843+0000 7fbf33251640 10 librados: watch_flush enter 2024-03-25T18:27:11.844 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.843+0000 7fbf33251640 10 librados: watch_flush exit 2024-03-25T18:27:11.845 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.843+0000 7fbf33251640 20 client.4713.objecter shutdown clearing up homeless session... 2024-03-25T18:27:11.845 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.843+0000 7fbf33251640 10 client.4713.objecter successfully canceled tick 2024-03-25T18:27:11.845 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.843+0000 7fbf33251640 1 -- 172.21.15.97:0/3991127513 >> v1:172.21.15.98:6800/800387573 conn(0x7fbf1006dc20 legacy=0x7fbf100700e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:11.845 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.843+0000 7fbf33251640 10 monclient: shutdown 2024-03-25T18:27:11.845 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.843+0000 7fbf33251640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:27:11.845 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.843+0000 7fbf33251640 1 -- 172.21.15.97:0/3991127513 >> v1:172.21.15.97:6789/0 conn(0x7fbf2c14be90 legacy=0x7fbf2c142bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:11.845 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.843+0000 7fbf33251640 1 -- 172.21.15.97:0/3991127513 shutdown_connections 2024-03-25T18:27:11.845 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.843+0000 7fbf33251640 1 -- 172.21.15.97:0/3991127513 >> 172.21.15.97:0/3991127513 conn(0x7fbf2c0617a0 msgr2=0x7fbf2c0b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:11.845 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.843+0000 7fbf33251640 1 -- 172.21.15.97:0/3991127513 shutdown_connections 2024-03-25T18:27:11.845 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.844+0000 7fbf33251640 1 -- 172.21.15.97:0/3991127513 wait complete. 2024-03-25T18:27:11.845 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:11.844+0000 7fbf33251640 1 librados: shutdown 2024-03-25T18:27:11.856 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:27:18.652 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:11:20.690 34931== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:18.659 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:11:20.698 34931== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:18.662 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:11:20.705 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:18.662 DEBUG:teuthology.orchestra.run.smithi097:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:27:18.666 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:11:20.704 34935== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:18.668 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:11:20.711 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:18.670 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:11:20.709 34935== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:18.687 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:11:20.725 34931== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:18.690 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:11:20.733 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:18.693 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:11:20.732 34935== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:18.758 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.756+0000 7f2a138a8640 10 monclient: get_monmap_and_config 2024-03-25T18:27:18.759 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.756+0000 7f2a138a8640 10 monclient: build_initial_monmap 2024-03-25T18:27:18.759 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.756+0000 7f2a138a8640 10 monclient: monmap: 2024-03-25T18:27:18.759 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:27:18.759 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:27:18.760 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:27:18.758202+0000 2024-03-25T18:27:18.760 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:27:18.758202+0000 2024-03-25T18:27:18.760 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:18.760 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:27:18.760 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:27:18.760 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T18:27:18.761 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-b 2024-03-25T18:27:18.761 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:27:18.761 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.757+0000 7f2a138a8640 1 Processor -- start 2024-03-25T18:27:18.763 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.757+0000 7f2a138a8640 1 -- start start 2024-03-25T18:27:18.763 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.757+0000 7f2a138a8640 10 monclient: init 2024-03-25T18:27:18.763 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.758+0000 7f2a138a8640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:18.764 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.758+0000 7f2a138a8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:27:18.764 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.758+0000 7f2a138a8640 10 monclient(hunting): picked mon.noname-c con 0x7f2a0c0b4d60 addr v1:172.21.15.98:6789/0 2024-03-25T18:27:18.764 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.758+0000 7f2a138a8640 10 monclient(hunting): picked mon.noname-b con 0x7f2a0c0b1730 addr v1:172.21.15.97:6790/0 2024-03-25T18:27:18.764 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.758+0000 7f2a138a8640 10 monclient(hunting): picked mon.noname-a con 0x7f2a0c14be90 addr v1:172.21.15.97:6789/0 2024-03-25T18:27:18.764 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.758+0000 7f2a138a8640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a0c0af940 con 0x7f2a0c14be90 2024-03-25T18:27:18.764 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.758+0000 7f2a138a8640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a0c0b0a70 con 0x7f2a0c0b4d60 2024-03-25T18:27:18.765 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.758+0000 7f2a138a8640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a0c0b0d50 con 0x7f2a0c0b1730 2024-03-25T18:27:18.765 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.758+0000 7f2a138a8640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:18.765 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.758+0000 7f2a138a8640 10 monclient(hunting): authenticate will time out at 8251.034596s 2024-03-25T18:27:18.783 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.781+0000 7f2a11644640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f2a0c0b4d60 0x7f2a0c13f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.97:42460/0 (socket says 172.21.15.97:42460) 2024-03-25T18:27:18.783 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.781+0000 7f2a11644640 1 -- 172.21.15.97:0/343139975 learned_addr learned my addr 172.21.15.97:0/343139975 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:27:18.825 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.823+0000 7f2a03fff640 1 -- 172.21.15.97:0/343139975 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 25021274 0 0) 0x7f2a0c0b0a70 con 0x7f2a0c0b4d60 2024-03-25T18:27:18.825 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.823+0000 7f2a03fff640 10 monclient(hunting): _negotiate 2024-03-25T18:27:18.825 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.823+0000 7f2a03fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:18.825 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.823+0000 7f2a03fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:18.825 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.823+0000 7f2a03fff640 10 monclient(hunting): my global_id is 4726 2024-03-25T18:27:18.825 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.823+0000 7f2a03fff640 1 -- 172.21.15.97:0/343139975 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29f8003520 con 0x7f2a0c0b4d60 2024-03-25T18:27:18.826 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.824+0000 7f2a03fff640 1 -- 172.21.15.97:0/343139975 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1654305997 0 0) 0x7f2a0c0b0d50 con 0x7f2a0c0b1730 2024-03-25T18:27:18.826 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.824+0000 7f2a03fff640 10 monclient(hunting): _negotiate 2024-03-25T18:27:18.826 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.824+0000 7f2a03fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:18.826 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.824+0000 7f2a03fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:18.826 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.824+0000 7f2a03fff640 10 monclient(hunting): my global_id is 4697 2024-03-25T18:27:18.827 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.824+0000 7f2a03fff640 1 -- 172.21.15.97:0/343139975 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a0c0b0a70 con 0x7f2a0c0b1730 2024-03-25T18:27:18.845 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.843+0000 7f2a03fff640 1 -- 172.21.15.97:0/343139975 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2779426746 0 0) 0x7f29f8003520 con 0x7f2a0c0b4d60 2024-03-25T18:27:18.845 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.843+0000 7f2a03fff640 1 -- 172.21.15.97:0/343139975 >> v1:172.21.15.97:6790/0 conn(0x7f2a0c0b1730 legacy=0x7f2a0c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:18.845 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.843+0000 7f2a03fff640 1 -- 172.21.15.97:0/343139975 >> v1:172.21.15.97:6789/0 conn(0x7f2a0c14be90 legacy=0x7f2a0c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:18.845 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.843+0000 7f2a03fff640 10 monclient: _finish_hunting 0 2024-03-25T18:27:18.845 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.843+0000 7f2a03fff640 1 monclient: found mon.noname-c 2024-03-25T18:27:18.845 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.843+0000 7f2a03fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:18.846 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.843+0000 7f2a03fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.98:6789/0 2024-03-25T18:27:18.846 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.843+0000 7f2a03fff640 1 -- 172.21.15.97:0/343139975 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a0c0b1030 con 0x7f2a0c0b4d60 2024-03-25T18:27:18.846 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.843+0000 7f2a03fff640 10 monclient: _finish_auth 0 2024-03-25T18:27:18.846 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.843+0000 7f2a03fff640 10 monclient: _check_auth_tickets 2024-03-25T18:27:18.846 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.843+0000 7f2a03fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:18.846 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.843+0000 7f2a138a8640 5 monclient: authenticate success, global_id 4726 2024-03-25T18:27:18.846 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.843+0000 7f2a138a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:27:18.857 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.855+0000 7f2a03fff640 1 -- 172.21.15.97:0/343139975 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f2a08001650 con 0x7f2a0c0b4d60 2024-03-25T18:27:18.857 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.855+0000 7f2a03fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:18.857 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.855+0000 7f2a03fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:27:18.857 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.855+0000 7f2a03fff640 10 monclient: dump: 2024-03-25T18:27:18.857 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:27:18.857 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:27:18.857 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:27:18.857 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:27:18.857 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:18.858 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:27:18.858 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:27:18.858 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:27:18.858 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:27:18.858 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:27:18.858 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.855+0000 7f2a138a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:27:18.871 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.869+0000 7f2a03fff640 1 -- 172.21.15.97:0/343139975 <== mon.1 v1:172.21.15.98:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2a080018a0 con 0x7f2a0c0b4d60 2024-03-25T18:27:18.871 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.869+0000 7f2a03fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:18.871 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.869+0000 7f2a03fff640 1 -- 172.21.15.97:0/343139975 <== mon.1 v1:172.21.15.98:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f2a08001c30 con 0x7f2a0c0b4d60 2024-03-25T18:27:18.871 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.869+0000 7f2a138a8640 10 monclient: get_monmap_and_config success 2024-03-25T18:27:18.871 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.869+0000 7f2a138a8640 4 set_mon_vals no callback set 2024-03-25T18:27:18.871 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.869+0000 7f2a03fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:18.871 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.869+0000 7f2a03fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:27:18.871 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.869+0000 7f2a03fff640 10 monclient: dump: 2024-03-25T18:27:18.871 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:27:18.871 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:27:18.871 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:27:18.872 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:27:18.872 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:18.872 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:27:18.872 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:27:18.872 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:27:18.872 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:27:18.872 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:27:18.873 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.870+0000 7f2a138a8640 10 monclient: shutdown 2024-03-25T18:27:18.873 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.870+0000 7f2a138a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:27:18.873 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.870+0000 7f2a138a8640 1 -- 172.21.15.97:0/343139975 >> v1:172.21.15.98:6789/0 conn(0x7f2a0c0b4d60 legacy=0x7f2a0c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:18.874 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.870+0000 7f2a138a8640 1 -- 172.21.15.97:0/343139975 shutdown_connections 2024-03-25T18:27:18.874 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.870+0000 7f2a138a8640 1 -- 172.21.15.97:0/343139975 >> 172.21.15.97:0/343139975 conn(0x7f2a0c0617a0 msgr2=0x7f2a0c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:18.874 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.870+0000 7f2a138a8640 1 -- 172.21.15.97:0/343139975 shutdown_connections 2024-03-25T18:27:18.874 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.870+0000 7f2a138a8640 1 -- 172.21.15.97:0/343139975 wait complete. 2024-03-25T18:27:18.874 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.871+0000 7f2a138a8640 10 monclient: build_initial_monmap 2024-03-25T18:27:18.874 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.871+0000 7f2a138a8640 10 monclient: monmap: 2024-03-25T18:27:18.874 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:27:18.874 INFO:teuthology.misc.health.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:27:18.874 INFO:teuthology.misc.health.smithi097.stderr:last_changed 0.000000 2024-03-25T18:27:18.874 INFO:teuthology.misc.health.smithi097.stderr:created 0.000000 2024-03-25T18:27:18.874 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:18.874 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 1 2024-03-25T18:27:18.875 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:27:18.875 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T18:27:18.875 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-c 2024-03-25T18:27:18.875 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:27:18.875 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.871+0000 7f2a138a8640 1 librados: starting msgr at 2024-03-25T18:27:18.875 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.871+0000 7f2a138a8640 1 librados: starting objecter 2024-03-25T18:27:18.875 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.871+0000 7f2a138a8640 1 Processor -- start 2024-03-25T18:27:18.875 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.871+0000 7f2a138a8640 1 -- start start 2024-03-25T18:27:18.875 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.871+0000 7f2a138a8640 1 librados: setting wanted keys 2024-03-25T18:27:18.875 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.871+0000 7f2a138a8640 1 librados: calling monclient init 2024-03-25T18:27:18.875 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.871+0000 7f2a138a8640 10 monclient: init 2024-03-25T18:27:18.876 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.872+0000 7f2a138a8640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:18.876 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.872+0000 7f2a138a8640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:27:18.876 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.872+0000 7f2a138a8640 10 monclient(hunting): picked mon.noname-b con 0x7f2a0c0b1730 addr v1:172.21.15.98:6789/0 2024-03-25T18:27:18.876 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.872+0000 7f2a138a8640 10 monclient(hunting): picked mon.noname-a con 0x7f2a0c0b4d60 addr v1:172.21.15.97:6789/0 2024-03-25T18:27:18.877 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.872+0000 7f2a138a8640 10 monclient(hunting): picked mon.noname-c con 0x7f2a0c14be90 addr v1:172.21.15.97:6790/0 2024-03-25T18:27:18.877 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.872+0000 7f2a138a8640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a0c13afa0 con 0x7f2a0c0b4d60 2024-03-25T18:27:18.877 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.872+0000 7f2a138a8640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a0c1399a0 con 0x7f2a0c0b1730 2024-03-25T18:27:18.877 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.872+0000 7f2a138a8640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a0c1432b0 con 0x7f2a0c14be90 2024-03-25T18:27:18.877 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.872+0000 7f2a138a8640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:18.877 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.872+0000 7f2a138a8640 10 monclient(hunting): authenticate will time out at 8251.148744s 2024-03-25T18:27:18.892 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.891+0000 7f2a10e43640 1 --1- >> v1:172.21.15.97:6789/0 conn(0x7f2a0c0b4d60 0x7f2a0c134fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.97:6789/0 says I am v1:172.21.15.97:32860/0 (socket says 172.21.15.97:32860) 2024-03-25T18:27:18.892 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.891+0000 7f2a10e43640 1 -- 172.21.15.97:0/3890848301 learned_addr learned my addr 172.21.15.97:0/3890848301 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:27:18.895 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.893+0000 7f2a01ffb640 10 client.?.objecter ms_handle_connect 0x7f2a0c0b4d60 2024-03-25T18:27:18.895 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.893+0000 7f2a01ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:18.895 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.894+0000 7f2a01ffb640 10 client.?.objecter ms_handle_connect 0x7f2a0c0b1730 2024-03-25T18:27:18.895 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.894+0000 7f2a01ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:18.899 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.897+0000 7f2a01ffb640 10 client.?.objecter ms_handle_connect 0x7f2a0c14be90 2024-03-25T18:27:18.899 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.897+0000 7f2a01ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:18.919 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.918+0000 7f2a01ffb640 1 -- 172.21.15.97:0/3890848301 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1955455526 0 0) 0x7f2a0c13afa0 con 0x7f2a0c0b4d60 2024-03-25T18:27:18.919 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.918+0000 7f2a01ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:27:18.920 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.918+0000 7f2a01ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:18.920 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.918+0000 7f2a01ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:18.920 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.918+0000 7f2a01ffb640 10 monclient(hunting): my global_id is 4719 2024-03-25T18:27:18.920 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.918+0000 7f2a01ffb640 1 -- 172.21.15.97:0/3890848301 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29f0004eb0 con 0x7f2a0c0b4d60 2024-03-25T18:27:18.920 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.919+0000 7f2a01ffb640 1 -- 172.21.15.97:0/3890848301 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1574245755 0 0) 0x7f2a0c1432b0 con 0x7f2a0c14be90 2024-03-25T18:27:18.920 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.919+0000 7f2a01ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:27:18.921 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.919+0000 7f2a01ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:18.921 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.919+0000 7f2a01ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:18.921 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.919+0000 7f2a01ffb640 10 monclient(hunting): my global_id is 4700 2024-03-25T18:27:18.921 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.919+0000 7f2a01ffb640 1 -- 172.21.15.97:0/3890848301 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a0c13afa0 con 0x7f2a0c14be90 2024-03-25T18:27:18.927 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.925+0000 7f2a01ffb640 1 -- 172.21.15.97:0/3890848301 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3470508485 0 0) 0x7f2a0c1399a0 con 0x7f2a0c0b1730 2024-03-25T18:27:18.927 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.925+0000 7f2a01ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:27:18.928 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.925+0000 7f2a01ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:18.928 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.925+0000 7f2a01ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:18.928 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.925+0000 7f2a01ffb640 10 monclient(hunting): my global_id is 4729 2024-03-25T18:27:18.928 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.925+0000 7f2a01ffb640 1 -- 172.21.15.97:0/3890848301 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a0c1432b0 con 0x7f2a0c0b1730 2024-03-25T18:27:18.940 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.938+0000 7f2a01ffb640 1 -- 172.21.15.97:0/3890848301 <== mon.2 v1:172.21.15.97:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3505560859 0 0) 0x7f2a0c13afa0 con 0x7f2a0c14be90 2024-03-25T18:27:18.940 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.938+0000 7f2a01ffb640 1 -- 172.21.15.97:0/3890848301 --> v1:172.21.15.97:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2a0c1399a0 con 0x7f2a0c14be90 2024-03-25T18:27:18.940 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.938+0000 7f2a01ffb640 1 -- 172.21.15.97:0/3890848301 <== mon.2 v1:172.21.15.97:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f2a04001b00 con 0x7f2a0c14be90 2024-03-25T18:27:18.940 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.938+0000 7f2a01ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:18.940 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.938+0000 7f2a01ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:27:18.940 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.938+0000 7f2a01ffb640 10 monclient(hunting): dump: 2024-03-25T18:27:18.940 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:27:18.940 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:27:18.940 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:27:18.940 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:27:18.940 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:18.941 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:27:18.941 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:27:18.941 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:27:18.941 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:27:18.941 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:27:18.941 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.938+0000 7f2a01ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:27:18.947 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.946+0000 7f2a01ffb640 1 -- 172.21.15.97:0/3890848301 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3163528999 0 0) 0x7f29f0004eb0 con 0x7f2a0c0b4d60 2024-03-25T18:27:18.947 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.946+0000 7f2a01ffb640 1 -- 172.21.15.97:0/3890848301 --> v1:172.21.15.97:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2a0c13afa0 con 0x7f2a0c0b4d60 2024-03-25T18:27:18.948 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.946+0000 7f2a01ffb640 1 -- 172.21.15.97:0/3890848301 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f29fc001a10 con 0x7f2a0c0b4d60 2024-03-25T18:27:18.948 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.946+0000 7f2a01ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:18.948 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.946+0000 7f2a01ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:27:18.948 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.946+0000 7f2a01ffb640 10 monclient(hunting): dump: 2024-03-25T18:27:18.948 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:27:18.948 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:27:18.948 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:27:18.948 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:27:18.948 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:18.948 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:27:18.948 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:27:18.948 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:27:18.948 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:27:18.949 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:27:18.949 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.948+0000 7f2a01ffb640 1 -- 172.21.15.97:0/3890848301 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2383341335 0 0) 0x7f2a0c1432b0 con 0x7f2a0c0b1730 2024-03-25T18:27:18.949 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.948+0000 7f2a01ffb640 1 -- 172.21.15.97:0/3890848301 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f29f0004eb0 con 0x7f2a0c0b1730 2024-03-25T18:27:18.950 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.948+0000 7f2a01ffb640 1 -- 172.21.15.97:0/3890848301 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f2a08001bb0 con 0x7f2a0c0b1730 2024-03-25T18:27:18.950 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.948+0000 7f2a01ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:18.950 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.948+0000 7f2a01ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:27:18.950 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.948+0000 7f2a01ffb640 10 monclient(hunting): dump: 2024-03-25T18:27:18.950 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:27:18.950 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:27:18.950 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:27:18.950 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:27:18.950 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:18.950 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:27:18.950 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:27:18.950 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:27:18.950 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:27:18.951 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:27:18.956 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.954+0000 7f2a01ffb640 1 -- 172.21.15.97:0/3890848301 <== mon.2 v1:172.21.15.97:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2572361447 0 0) 0x7f2a0c1399a0 con 0x7f2a0c14be90 2024-03-25T18:27:18.957 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.954+0000 7f2a01ffb640 1 -- 172.21.15.97:0/3890848301 >> v1:172.21.15.98:6789/0 conn(0x7f2a0c0b1730 legacy=0x7f2a0c139290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:18.957 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.954+0000 7f2a01ffb640 1 -- 172.21.15.97:0/3890848301 >> v1:172.21.15.97:6789/0 conn(0x7f2a0c0b4d60 legacy=0x7f2a0c134fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:18.957 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.954+0000 7f2a01ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:27:18.957 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.954+0000 7f2a01ffb640 1 monclient: found mon.c 2024-03-25T18:27:18.957 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.954+0000 7f2a01ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:18.957 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.954+0000 7f2a01ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.97:6790/0 2024-03-25T18:27:18.957 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.954+0000 7f2a01ffb640 1 -- 172.21.15.97:0/3890848301 --> v1:172.21.15.97:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a0c1d50c0 con 0x7f2a0c14be90 2024-03-25T18:27:18.957 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.954+0000 7f2a01ffb640 10 monclient: _finish_auth 0 2024-03-25T18:27:18.957 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.954+0000 7f2a01ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:27:18.957 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.954+0000 7f2a01ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:18.957 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.954+0000 7f2a138a8640 5 monclient: authenticate success, global_id 4700 2024-03-25T18:27:18.958 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.954+0000 7f2a138a8640 10 monclient: _renew_subs 2024-03-25T18:27:18.958 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.954+0000 7f2a138a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.97:6790/0 2024-03-25T18:27:18.958 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.954+0000 7f2a138a8640 1 -- 172.21.15.97:0/3890848301 --> v1:172.21.15.97:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a0c1356f0 con 0x7f2a0c14be90 2024-03-25T18:27:18.958 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.955+0000 7f2a138a8640 10 client.4700.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:27:18.958 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.955+0000 7f2a138a8640 10 monclient: _renew_subs 2024-03-25T18:27:18.958 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.955+0000 7f2a138a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.97:6790/0 2024-03-25T18:27:18.958 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.955+0000 7f2a138a8640 1 -- 172.21.15.97:0/3890848301 --> v1:172.21.15.97:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a0c1d80b0 con 0x7f2a0c14be90 2024-03-25T18:27:18.958 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.955+0000 7f2a138a8640 1 librados: init done 2024-03-25T18:27:18.959 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.956+0000 7f2a138a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:18.959 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.956+0000 7f2a138a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:18.959 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.956+0000 7f2a138a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.97:6790/0 2024-03-25T18:27:18.959 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:18.956+0000 7f2a138a8640 1 -- 172.21.15.97:0/3890848301 --> v1:172.21.15.97:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a0c1d7610 con 0x7f2a0c14be90 2024-03-25T18:27:19.256 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:19.254+0000 7f2a01ffb640 1 -- 172.21.15.97:0/3890848301 <== mon.2 v1:172.21.15.97:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2a04001fb0 con 0x7f2a0c14be90 2024-03-25T18:27:19.256 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:19.254+0000 7f2a01ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:19.256 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:19.254+0000 7f2a01ffb640 1 -- 172.21.15.97:0/3890848301 <== mon.2 v1:172.21.15.97:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f2a04003b10 con 0x7f2a0c14be90 2024-03-25T18:27:19.256 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:19.254+0000 7f2a01ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:19.256 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:19.254+0000 7f2a01ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:27:19.256 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:19.254+0000 7f2a01ffb640 10 monclient: dump: 2024-03-25T18:27:19.256 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:27:19.257 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:27:19.257 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:27:19.257 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:27:19.257 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:19.257 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:27:19.257 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:27:19.257 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:27:19.257 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:27:19.257 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:27:19.257 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:19.254+0000 7f2a037fe640 4 set_mon_vals no callback set 2024-03-25T18:27:19.258 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:19.255+0000 7f2a01ffb640 1 -- 172.21.15.97:0/3890848301 <== mon.2 v1:172.21.15.97:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2377622137 0 0) 0x7f2a04019ae0 con 0x7f2a0c14be90 2024-03-25T18:27:19.258 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:19.256+0000 7f2a11644640 10 monclient: get_auth_request con 0x7f29f006e0b0 auth_method 0 2024-03-25T18:27:19.258 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:19.256+0000 7f2a01ffb640 10 client.4700.objecter ms_handle_connect 0x7f29f006e0b0 2024-03-25T18:27:19.262 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:19.260+0000 7f2a01ffb640 1 -- 172.21.15.97:0/3890848301 <== mon.2 v1:172.21.15.97:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3078631805 0 0) 0x7f2a040536f0 con 0x7f2a0c14be90 2024-03-25T18:27:19.262 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:19.260+0000 7f2a01ffb640 10 client.4700.objecter ms_dispatch 0x7f2a0c000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:27:19.262 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:19.260+0000 7f2a01ffb640 3 client.4700.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:27:19.262 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:19.260+0000 7f2a01ffb640 3 client.4700.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:27:19.263 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:19.260+0000 7f2a01ffb640 20 client.4700.objecter dump_active .. 0 homeless 2024-03-25T18:27:19.263 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:19.260+0000 7f2a01ffb640 1 -- 172.21.15.97:0/3890848301 <== mon.2 v1:172.21.15.97:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2a0404f030 con 0x7f2a0c14be90 2024-03-25T18:27:19.263 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:19.260+0000 7f2a01ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:19.263 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:19.260+0000 7f2a01ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:27:19.504 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:19.502+0000 7f2a138a8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:19.504 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:19.502+0000 7f2a138a8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:19.504 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:19.502+0000 7f2a138a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.97:6790/0 2024-03-25T18:27:19.504 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:19.502+0000 7f2a138a8640 1 -- 172.21.15.97:0/3890848301 --> v1:172.21.15.97:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2a0c0b2270 con 0x7f2a0c14be90 2024-03-25T18:27:19.528 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:19.526+0000 7f2a01ffb640 1 -- 172.21.15.97:0/3890848301 <== mon.2 v1:172.21.15.97:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 337131954) 0x7f2a04051130 con 0x7f2a0c14be90 2024-03-25T18:27:19.529 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:19.526+0000 7f2a01ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:19.529 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:19.526+0000 7f2a01ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:27:19.529 INFO:teuthology.misc.health.smithi097.stdout: 2024-03-25T18:27:19.529 INFO:teuthology.misc.health.smithi097.stdout:{"epoch":11,"fsid":"4c3d472c-79b8-486b-9e3d-2738d4306754","created":"2024-03-25T18:16:14.737309+0000","modified":"2024-03-25T18:26:18.750062+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:18.750062+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cb5152b9-262f-45c3-94b8-18d0e81df558","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"6daf8793-0838-473d-a7d6-9a0d2894c75d","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"136f1493-3b9c-4f43-a2e7-641482d69dae","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"397d538c-4f64-4daf-9e8b-662d09f91327","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4f1df08-663f-4bda-93f8-3878ab7c24b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c4b18d70-a3ce-4b15-9b70-eae5525bd761","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8dd4033a-de0e-4f8a-96dc-59f3c1f825b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0077722e-ffa6-4a3d-9891-929d9869b286","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:27:19.531 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:19.529+0000 7f2a138a8640 10 librados: watch_flush enter 2024-03-25T18:27:19.531 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:19.529+0000 7f2a138a8640 10 librados: watch_flush exit 2024-03-25T18:27:19.531 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:19.529+0000 7f2a138a8640 20 client.4700.objecter shutdown clearing up homeless session... 2024-03-25T18:27:19.531 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:19.529+0000 7f2a138a8640 10 client.4700.objecter successfully canceled tick 2024-03-25T18:27:19.531 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:19.529+0000 7f2a138a8640 1 -- 172.21.15.97:0/3890848301 >> v1:172.21.15.98:6800/800387573 conn(0x7f29f006e0b0 legacy=0x7f29f0070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:19.532 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:19.529+0000 7f2a138a8640 10 monclient: shutdown 2024-03-25T18:27:19.532 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:19.529+0000 7f2a138a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:27:19.532 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:19.529+0000 7f2a138a8640 1 -- 172.21.15.97:0/3890848301 >> v1:172.21.15.97:6790/0 conn(0x7f2a0c14be90 legacy=0x7f2a0c1d77c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:19.532 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:19.529+0000 7f2a138a8640 1 -- 172.21.15.97:0/3890848301 shutdown_connections 2024-03-25T18:27:19.532 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:19.529+0000 7f2a138a8640 1 -- 172.21.15.97:0/3890848301 >> 172.21.15.97:0/3890848301 conn(0x7f2a0c0617a0 msgr2=0x7f2a0c0b6ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:19.532 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:19.529+0000 7f2a138a8640 1 -- 172.21.15.97:0/3890848301 shutdown_connections 2024-03-25T18:27:19.532 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:19.529+0000 7f2a138a8640 1 -- 172.21.15.97:0/3890848301 wait complete. 2024-03-25T18:27:19.532 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:19.529+0000 7f2a138a8640 1 librados: shutdown 2024-03-25T18:27:19.544 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:27:24.084 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:11:26.122 34931== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:24.129 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:11:26.172 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:24.131 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:11:26.169 34935== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:24.133 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:11:26.177 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:24.136 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:11:26.175 34931== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:24.142 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:11:26.181 34935== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:24.170 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:11:26.213 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:24.172 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:11:26.211 34935== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:24.175 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:11:26.214 34931== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:26.352 DEBUG:teuthology.orchestra.run.smithi097:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:27:26.456 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.454+0000 7fe524b6d640 10 monclient: get_monmap_and_config 2024-03-25T18:27:26.456 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.454+0000 7fe524b6d640 10 monclient: build_initial_monmap 2024-03-25T18:27:26.456 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.454+0000 7fe524b6d640 10 monclient: monmap: 2024-03-25T18:27:26.456 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:27:26.456 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:27:26.457 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:27:26.455872+0000 2024-03-25T18:27:26.457 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:27:26.455872+0000 2024-03-25T18:27:26.457 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:26.457 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:27:26.457 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:27:26.457 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T18:27:26.457 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-b 2024-03-25T18:27:26.457 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:27:26.461 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.455+0000 7fe524b6d640 1 Processor -- start 2024-03-25T18:27:26.461 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.455+0000 7fe524b6d640 1 -- start start 2024-03-25T18:27:26.461 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.455+0000 7fe524b6d640 10 monclient: init 2024-03-25T18:27:26.461 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.456+0000 7fe524b6d640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:26.461 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.456+0000 7fe524b6d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:27:26.461 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.456+0000 7fe524b6d640 10 monclient(hunting): picked mon.noname-b con 0x7fe5200a98f0 addr v1:172.21.15.97:6790/0 2024-03-25T18:27:26.461 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.456+0000 7fe524b6d640 10 monclient(hunting): picked mon.noname-c con 0x7fe52013f710 addr v1:172.21.15.98:6789/0 2024-03-25T18:27:26.462 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.456+0000 7fe524b6d640 10 monclient(hunting): picked mon.noname-a con 0x7fe5201430c0 addr v1:172.21.15.97:6789/0 2024-03-25T18:27:26.462 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.456+0000 7fe524b6d640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe520142520 con 0x7fe5201430c0 2024-03-25T18:27:26.462 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.456+0000 7fe524b6d640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5200b28a0 con 0x7fe52013f710 2024-03-25T18:27:26.462 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.456+0000 7fe524b6d640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5200b2a30 con 0x7fe5200a98f0 2024-03-25T18:27:26.462 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.456+0000 7fe524b6d640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:26.462 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.456+0000 7fe524b6d640 10 monclient(hunting): authenticate will time out at 8258.732364s 2024-03-25T18:27:26.462 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.458+0000 7fe51dd9b640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fe52013f710 0x7fe5201429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.97:50334/0 (socket says 172.21.15.97:50334) 2024-03-25T18:27:26.462 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.458+0000 7fe51dd9b640 1 -- 172.21.15.97:0/2650933688 learned_addr learned my addr 172.21.15.97:0/2650933688 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:27:26.491 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.489+0000 7fe51d59a640 1 -- 172.21.15.97:0/2650933688 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2640219865 0 0) 0x7fe5200b2a30 con 0x7fe5200a98f0 2024-03-25T18:27:26.491 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.489+0000 7fe51d59a640 10 monclient(hunting): _negotiate 2024-03-25T18:27:26.491 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.489+0000 7fe51d59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:26.491 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.489+0000 7fe51d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:26.491 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.489+0000 7fe51d59a640 10 monclient(hunting): my global_id is 4703 2024-03-25T18:27:26.491 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.489+0000 7fe51d59a640 1 -- 172.21.15.97:0/2650933688 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe508003520 con 0x7fe5200a98f0 2024-03-25T18:27:26.496 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.495+0000 7fe51d59a640 1 -- 172.21.15.97:0/2650933688 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 545582669 0 0) 0x7fe520142520 con 0x7fe5201430c0 2024-03-25T18:27:26.496 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.495+0000 7fe51d59a640 10 monclient(hunting): _negotiate 2024-03-25T18:27:26.496 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.495+0000 7fe51d59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:26.496 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.495+0000 7fe51d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:26.497 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.495+0000 7fe51d59a640 10 monclient(hunting): my global_id is 4722 2024-03-25T18:27:26.497 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.495+0000 7fe51d59a640 1 -- 172.21.15.97:0/2650933688 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5200b2a30 con 0x7fe5201430c0 2024-03-25T18:27:26.508 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.506+0000 7fe51d59a640 1 -- 172.21.15.97:0/2650933688 <== mon.2 v1:172.21.15.97:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1708167813 0 0) 0x7fe508003520 con 0x7fe5200a98f0 2024-03-25T18:27:26.508 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.506+0000 7fe51d59a640 1 -- 172.21.15.97:0/2650933688 >> v1:172.21.15.98:6789/0 conn(0x7fe52013f710 legacy=0x7fe5201429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:26.508 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.506+0000 7fe51d59a640 1 -- 172.21.15.97:0/2650933688 >> v1:172.21.15.97:6789/0 conn(0x7fe5201430c0 legacy=0x7fe52014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:26.508 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.506+0000 7fe51d59a640 10 monclient: _finish_hunting 0 2024-03-25T18:27:26.508 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.506+0000 7fe51d59a640 1 monclient: found mon.noname-b 2024-03-25T18:27:26.508 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.506+0000 7fe51d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:26.508 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.506+0000 7fe51d59a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.97:6790/0 2024-03-25T18:27:26.508 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.506+0000 7fe51d59a640 1 -- 172.21.15.97:0/2650933688 --> v1:172.21.15.97:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5200b44c0 con 0x7fe5200a98f0 2024-03-25T18:27:26.508 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.506+0000 7fe51d59a640 10 monclient: _finish_auth 0 2024-03-25T18:27:26.508 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.506+0000 7fe51d59a640 10 monclient: _check_auth_tickets 2024-03-25T18:27:26.508 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.506+0000 7fe51d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:26.508 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.506+0000 7fe51d59a640 1 -- 172.21.15.97:0/2650933688 <== mon.2 v1:172.21.15.97:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7fe510001690 con 0x7fe5200a98f0 2024-03-25T18:27:26.508 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.506+0000 7fe51d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:26.509 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.506+0000 7fe51d59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:27:26.509 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.506+0000 7fe51d59a640 10 monclient: dump: 2024-03-25T18:27:26.509 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:27:26.509 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:27:26.509 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:27:26.509 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:27:26.509 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:26.509 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:27:26.509 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:27:26.509 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:27:26.509 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:27:26.509 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:27:26.509 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.506+0000 7fe524b6d640 5 monclient: authenticate success, global_id 4703 2024-03-25T18:27:26.509 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.506+0000 7fe524b6d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:27:26.517 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.515+0000 7fe51d59a640 1 -- 172.21.15.97:0/2650933688 <== mon.2 v1:172.21.15.97:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe5100017f0 con 0x7fe5200a98f0 2024-03-25T18:27:26.517 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.515+0000 7fe51d59a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:26.517 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.515+0000 7fe51d59a640 1 -- 172.21.15.97:0/2650933688 <== mon.2 v1:172.21.15.97:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7fe510001c00 con 0x7fe5200a98f0 2024-03-25T18:27:26.517 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.515+0000 7fe51d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:26.517 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.515+0000 7fe51d59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:27:26.517 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.515+0000 7fe51d59a640 10 monclient: dump: 2024-03-25T18:27:26.517 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:27:26.517 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:27:26.517 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:27:26.517 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:27:26.517 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:26.517 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:27:26.517 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:27:26.517 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:27:26.518 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:27:26.518 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:27:26.518 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.515+0000 7fe524b6d640 10 monclient: get_monmap_and_config success 2024-03-25T18:27:26.518 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.515+0000 7fe524b6d640 4 set_mon_vals no callback set 2024-03-25T18:27:26.520 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.516+0000 7fe524b6d640 10 monclient: shutdown 2024-03-25T18:27:26.520 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.516+0000 7fe524b6d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:27:26.520 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.516+0000 7fe524b6d640 1 -- 172.21.15.97:0/2650933688 >> v1:172.21.15.97:6790/0 conn(0x7fe5200a98f0 legacy=0x7fe5200a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:26.520 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.516+0000 7fe524b6d640 1 -- 172.21.15.97:0/2650933688 shutdown_connections 2024-03-25T18:27:26.520 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.516+0000 7fe524b6d640 1 -- 172.21.15.97:0/2650933688 >> 172.21.15.97:0/2650933688 conn(0x7fe5200b7ea0 msgr2=0x7fe5200b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:26.520 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.516+0000 7fe524b6d640 1 -- 172.21.15.97:0/2650933688 shutdown_connections 2024-03-25T18:27:26.520 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.516+0000 7fe524b6d640 1 -- 172.21.15.97:0/2650933688 wait complete. 2024-03-25T18:27:26.521 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.517+0000 7fe524b6d640 10 monclient: build_initial_monmap 2024-03-25T18:27:26.521 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.517+0000 7fe524b6d640 10 monclient: monmap: 2024-03-25T18:27:26.521 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:27:26.521 INFO:teuthology.misc.health.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:27:26.521 INFO:teuthology.misc.health.smithi097.stderr:last_changed 0.000000 2024-03-25T18:27:26.521 INFO:teuthology.misc.health.smithi097.stderr:created 0.000000 2024-03-25T18:27:26.521 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:26.521 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 1 2024-03-25T18:27:26.521 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:27:26.521 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T18:27:26.521 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-c 2024-03-25T18:27:26.521 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:27:26.521 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.517+0000 7fe524b6d640 1 librados: starting msgr at 2024-03-25T18:27:26.521 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.517+0000 7fe524b6d640 1 librados: starting objecter 2024-03-25T18:27:26.521 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.517+0000 7fe524b6d640 1 Processor -- start 2024-03-25T18:27:26.521 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.517+0000 7fe524b6d640 1 -- start start 2024-03-25T18:27:26.522 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.517+0000 7fe524b6d640 1 librados: setting wanted keys 2024-03-25T18:27:26.522 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.517+0000 7fe524b6d640 1 librados: calling monclient init 2024-03-25T18:27:26.522 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.517+0000 7fe524b6d640 10 monclient: init 2024-03-25T18:27:26.522 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.518+0000 7fe524b6d640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:26.522 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.518+0000 7fe524b6d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:27:26.522 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.518+0000 7fe524b6d640 10 monclient(hunting): picked mon.noname-b con 0x7fe5200a98f0 addr v1:172.21.15.98:6789/0 2024-03-25T18:27:26.522 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.518+0000 7fe524b6d640 10 monclient(hunting): picked mon.noname-c con 0x7fe52013f710 addr v1:172.21.15.97:6790/0 2024-03-25T18:27:26.522 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.518+0000 7fe524b6d640 10 monclient(hunting): picked mon.noname-a con 0x7fe5201430c0 addr v1:172.21.15.97:6789/0 2024-03-25T18:27:26.522 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.518+0000 7fe524b6d640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5201370a0 con 0x7fe5201430c0 2024-03-25T18:27:26.522 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.518+0000 7fe524b6d640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe520135090 con 0x7fe5200a98f0 2024-03-25T18:27:26.522 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.518+0000 7fe524b6d640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe520135220 con 0x7fe52013f710 2024-03-25T18:27:26.522 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.518+0000 7fe524b6d640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:26.522 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.518+0000 7fe524b6d640 10 monclient(hunting): authenticate will time out at 8258.794251s 2024-03-25T18:27:26.526 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.525+0000 7fe51e59c640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fe5200a98f0 0x7fe5201dae50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.97:50376/0 (socket says 172.21.15.97:50376) 2024-03-25T18:27:26.526 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.525+0000 7fe51e59c640 1 -- 172.21.15.97:0/225116119 learned_addr learned my addr 172.21.15.97:0/225116119 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:27:26.530 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.528+0000 7fe50effd640 10 client.?.objecter ms_handle_connect 0x7fe5200a98f0 2024-03-25T18:27:26.530 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.528+0000 7fe50effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:26.535 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.534+0000 7fe50effd640 10 client.?.objecter ms_handle_connect 0x7fe52013f710 2024-03-25T18:27:26.535 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.534+0000 7fe50effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:26.537 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.536+0000 7fe50effd640 10 client.?.objecter ms_handle_connect 0x7fe5201430c0 2024-03-25T18:27:26.537 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.536+0000 7fe50effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:26.539 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.537+0000 7fe50effd640 1 -- 172.21.15.97:0/225116119 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3898991728 0 0) 0x7fe520135090 con 0x7fe5200a98f0 2024-03-25T18:27:26.539 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.537+0000 7fe50effd640 10 monclient(hunting): _negotiate 2024-03-25T18:27:26.539 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.537+0000 7fe50effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:26.539 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.537+0000 7fe50effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:26.539 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.537+0000 7fe50effd640 10 monclient(hunting): my global_id is 4735 2024-03-25T18:27:26.539 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.537+0000 7fe50effd640 1 -- 172.21.15.97:0/225116119 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe500004eb0 con 0x7fe5200a98f0 2024-03-25T18:27:26.546 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.544+0000 7fe50effd640 1 -- 172.21.15.97:0/225116119 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2434828332 0 0) 0x7fe500004eb0 con 0x7fe5200a98f0 2024-03-25T18:27:26.546 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.544+0000 7fe50effd640 1 -- 172.21.15.97:0/225116119 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe520135090 con 0x7fe5200a98f0 2024-03-25T18:27:26.546 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.544+0000 7fe50effd640 1 -- 172.21.15.97:0/225116119 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7fe510001690 con 0x7fe5200a98f0 2024-03-25T18:27:26.546 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.544+0000 7fe50effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:26.546 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.544+0000 7fe50effd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:27:26.546 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.544+0000 7fe50effd640 10 monclient(hunting): dump: 2024-03-25T18:27:26.546 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:27:26.546 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:27:26.546 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:27:26.546 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:27:26.547 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:26.547 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:27:26.547 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:27:26.547 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:27:26.547 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:27:26.547 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:27:26.547 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.544+0000 7fe50effd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:27:26.551 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.549+0000 7fe50effd640 1 -- 172.21.15.97:0/225116119 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1700269921 0 0) 0x7fe5201370a0 con 0x7fe5201430c0 2024-03-25T18:27:26.551 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.549+0000 7fe50effd640 10 monclient(hunting): _negotiate 2024-03-25T18:27:26.551 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.549+0000 7fe50effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:26.551 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.549+0000 7fe50effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:26.551 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.549+0000 7fe50effd640 10 monclient(hunting): my global_id is 4725 2024-03-25T18:27:26.551 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.549+0000 7fe50effd640 1 -- 172.21.15.97:0/225116119 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe500004eb0 con 0x7fe5201430c0 2024-03-25T18:27:26.554 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.552+0000 7fe50effd640 1 -- 172.21.15.97:0/225116119 <== mon.1 v1:172.21.15.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 245354668 0 0) 0x7fe520135090 con 0x7fe5200a98f0 2024-03-25T18:27:26.554 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.552+0000 7fe50effd640 1 -- 172.21.15.97:0/225116119 >> v1:172.21.15.97:6790/0 conn(0x7fe52013f710 legacy=0x7fe5201d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:26.554 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.552+0000 7fe50effd640 1 -- 172.21.15.97:0/225116119 >> v1:172.21.15.97:6789/0 conn(0x7fe5201430c0 legacy=0x7fe5201d48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:26.554 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.552+0000 7fe50effd640 10 monclient: _finish_hunting 0 2024-03-25T18:27:26.554 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.552+0000 7fe50effd640 1 monclient: found mon.b 2024-03-25T18:27:26.555 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.552+0000 7fe50effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:26.555 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.552+0000 7fe50effd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:27:26.555 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.552+0000 7fe50effd640 1 -- 172.21.15.97:0/225116119 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5201d5000 con 0x7fe5200a98f0 2024-03-25T18:27:26.555 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.552+0000 7fe50effd640 10 monclient: _finish_auth 0 2024-03-25T18:27:26.555 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.552+0000 7fe50effd640 10 monclient: _check_auth_tickets 2024-03-25T18:27:26.555 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.552+0000 7fe50effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:26.555 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.552+0000 7fe524b6d640 5 monclient: authenticate success, global_id 4735 2024-03-25T18:27:26.555 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.552+0000 7fe524b6d640 10 monclient: _renew_subs 2024-03-25T18:27:26.555 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.552+0000 7fe524b6d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:27:26.555 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.552+0000 7fe524b6d640 1 -- 172.21.15.97:0/225116119 --> v1:172.21.15.98:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5201d52b0 con 0x7fe5200a98f0 2024-03-25T18:27:26.555 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.552+0000 7fe524b6d640 10 client.4735.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:27:26.555 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.552+0000 7fe524b6d640 10 monclient: _renew_subs 2024-03-25T18:27:26.555 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.552+0000 7fe524b6d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:27:26.556 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.552+0000 7fe524b6d640 1 -- 172.21.15.97:0/225116119 --> v1:172.21.15.98:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe520135910 con 0x7fe5200a98f0 2024-03-25T18:27:26.556 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.552+0000 7fe524b6d640 1 librados: init done 2024-03-25T18:27:26.556 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.553+0000 7fe524b6d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:26.556 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.553+0000 7fe524b6d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:26.556 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.553+0000 7fe524b6d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:27:26.556 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.553+0000 7fe524b6d640 1 -- 172.21.15.97:0/225116119 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5200627b0 con 0x7fe5200a98f0 2024-03-25T18:27:26.559 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.558+0000 7fe50effd640 1 -- 172.21.15.97:0/225116119 <== mon.1 v1:172.21.15.98:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe5100038a0 con 0x7fe5200a98f0 2024-03-25T18:27:26.560 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.558+0000 7fe50effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:26.560 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.558+0000 7fe51cd99640 4 set_mon_vals no callback set 2024-03-25T18:27:26.830 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.828+0000 7fe50effd640 1 -- 172.21.15.97:0/225116119 <== mon.1 v1:172.21.15.98:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7fe510003cf0 con 0x7fe5200a98f0 2024-03-25T18:27:26.830 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.828+0000 7fe50effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:26.830 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.828+0000 7fe50effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:27:26.830 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.828+0000 7fe50effd640 10 monclient: dump: 2024-03-25T18:27:26.830 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:27:26.830 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:27:26.830 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:27:26.830 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:27:26.830 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:26.831 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:27:26.831 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:27:26.831 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:27:26.831 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:27:26.831 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:27:26.831 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.829+0000 7fe50effd640 1 -- 172.21.15.97:0/225116119 <== mon.1 v1:172.21.15.98:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2377622137 0 0) 0x7fe510003ed0 con 0x7fe5200a98f0 2024-03-25T18:27:26.831 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.830+0000 7fe51dd9b640 10 monclient: get_auth_request con 0x7fe50006dd90 auth_method 0 2024-03-25T18:27:26.832 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.830+0000 7fe50effd640 10 client.4735.objecter ms_handle_connect 0x7fe50006dd90 2024-03-25T18:27:26.834 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.833+0000 7fe50effd640 1 -- 172.21.15.97:0/225116119 <== mon.1 v1:172.21.15.98:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3078631805 0 0) 0x7fe5100549a0 con 0x7fe5200a98f0 2024-03-25T18:27:26.834 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.833+0000 7fe50effd640 10 client.4735.objecter ms_dispatch 0x7fe520000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:27:26.835 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.833+0000 7fe50effd640 3 client.4735.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:27:26.835 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.833+0000 7fe50effd640 3 client.4735.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:27:26.835 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.833+0000 7fe50effd640 20 client.4735.objecter dump_active .. 0 homeless 2024-03-25T18:27:26.837 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.836+0000 7fe50effd640 1 -- 172.21.15.97:0/225116119 <== mon.1 v1:172.21.15.98:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe5100504b0 con 0x7fe5200a98f0 2024-03-25T18:27:26.837 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.836+0000 7fe50effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:26.838 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:26.836+0000 7fe50effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:27:27.076 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:27.074+0000 7fe524b6d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:27.077 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:27.074+0000 7fe524b6d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:27.077 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:27.074+0000 7fe524b6d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T18:27:27.077 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:27.074+0000 7fe524b6d640 1 -- 172.21.15.97:0/225116119 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe52013bfe0 con 0x7fe5200a98f0 2024-03-25T18:27:27.100 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:27.098+0000 7fe50effd640 1 -- 172.21.15.97:0/225116119 <== mon.1 v1:172.21.15.98:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 337131954) 0x7fe5100523e0 con 0x7fe5200a98f0 2024-03-25T18:27:27.100 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:27.098+0000 7fe50effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:27.100 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:27.098+0000 7fe50effd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:27:27.100 INFO:teuthology.misc.health.smithi097.stdout: 2024-03-25T18:27:27.100 INFO:teuthology.misc.health.smithi097.stdout:{"epoch":11,"fsid":"4c3d472c-79b8-486b-9e3d-2738d4306754","created":"2024-03-25T18:16:14.737309+0000","modified":"2024-03-25T18:26:18.750062+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:18.750062+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cb5152b9-262f-45c3-94b8-18d0e81df558","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"6daf8793-0838-473d-a7d6-9a0d2894c75d","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"136f1493-3b9c-4f43-a2e7-641482d69dae","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"397d538c-4f64-4daf-9e8b-662d09f91327","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4f1df08-663f-4bda-93f8-3878ab7c24b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c4b18d70-a3ce-4b15-9b70-eae5525bd761","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8dd4033a-de0e-4f8a-96dc-59f3c1f825b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0077722e-ffa6-4a3d-9891-929d9869b286","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:27:27.103 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:27.101+0000 7fe524b6d640 10 librados: watch_flush enter 2024-03-25T18:27:27.103 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:27.101+0000 7fe524b6d640 10 librados: watch_flush exit 2024-03-25T18:27:27.103 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:27.101+0000 7fe524b6d640 20 client.4735.objecter shutdown clearing up homeless session... 2024-03-25T18:27:27.103 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:27.101+0000 7fe524b6d640 10 client.4735.objecter successfully canceled tick 2024-03-25T18:27:27.103 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:27.101+0000 7fe524b6d640 1 -- 172.21.15.97:0/225116119 >> v1:172.21.15.98:6800/800387573 conn(0x7fe50006dd90 legacy=0x7fe500070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:27.103 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:27.101+0000 7fe524b6d640 10 monclient: shutdown 2024-03-25T18:27:27.103 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:27.101+0000 7fe524b6d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:27:27.103 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:27.101+0000 7fe524b6d640 1 -- 172.21.15.97:0/225116119 >> v1:172.21.15.98:6789/0 conn(0x7fe5200a98f0 legacy=0x7fe5201dae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:27.103 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:27.101+0000 7fe524b6d640 1 -- 172.21.15.97:0/225116119 shutdown_connections 2024-03-25T18:27:27.103 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:27.101+0000 7fe524b6d640 1 -- 172.21.15.97:0/225116119 >> 172.21.15.97:0/225116119 conn(0x7fe5200b7ea0 msgr2=0x7fe520074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:27.104 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:27.101+0000 7fe524b6d640 1 -- 172.21.15.97:0/225116119 shutdown_connections 2024-03-25T18:27:27.104 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:27.101+0000 7fe524b6d640 1 -- 172.21.15.97:0/225116119 wait complete. 2024-03-25T18:27:27.104 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:27.101+0000 7fe524b6d640 1 librados: shutdown 2024-03-25T18:27:27.115 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:27:33.921 DEBUG:teuthology.orchestra.run.smithi097:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:27:34.019 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.017+0000 7fed2d87d640 10 monclient: get_monmap_and_config 2024-03-25T18:27:34.019 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.017+0000 7fed2d87d640 10 monclient: build_initial_monmap 2024-03-25T18:27:34.019 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.017+0000 7fed2d87d640 10 monclient: monmap: 2024-03-25T18:27:34.019 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:27:34.019 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:27:34.019 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:27:34.018520+0000 2024-03-25T18:27:34.019 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:27:34.018520+0000 2024-03-25T18:27:34.019 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:34.019 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:27:34.019 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:27:34.020 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T18:27:34.020 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-b 2024-03-25T18:27:34.020 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:27:34.021 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.017+0000 7fed2d87d640 1 Processor -- start 2024-03-25T18:27:34.021 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.017+0000 7fed2d87d640 1 -- start start 2024-03-25T18:27:34.021 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.017+0000 7fed2d87d640 10 monclient: init 2024-03-25T18:27:34.021 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.018+0000 7fed2d87d640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:34.021 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.018+0000 7fed2d87d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:27:34.021 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.018+0000 7fed2d87d640 10 monclient(hunting): picked mon.noname-a con 0x7fed280b4d60 addr v1:172.21.15.97:6789/0 2024-03-25T18:27:34.021 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.018+0000 7fed2d87d640 10 monclient(hunting): picked mon.noname-c con 0x7fed280b1730 addr v1:172.21.15.98:6789/0 2024-03-25T18:27:34.021 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.018+0000 7fed2d87d640 10 monclient(hunting): picked mon.noname-b con 0x7fed2814be90 addr v1:172.21.15.97:6790/0 2024-03-25T18:27:34.022 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.018+0000 7fed2d87d640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed280af940 con 0x7fed280b4d60 2024-03-25T18:27:34.022 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.018+0000 7fed2d87d640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed280b0a70 con 0x7fed280b1730 2024-03-25T18:27:34.022 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.018+0000 7fed2d87d640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed280b0d50 con 0x7fed2814be90 2024-03-25T18:27:34.022 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.018+0000 7fed2d87d640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:34.022 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.018+0000 7fed2d87d640 10 monclient(hunting): authenticate will time out at 8266.294397s 2024-03-25T18:27:34.022 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.018+0000 7fed277fe640 1 --1- >> v1:172.21.15.97:6790/0 conn(0x7fed2814be90 0x7fed2814e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.97:6790/0 says I am v1:172.21.15.97:57774/0 (socket says 172.21.15.97:57774) 2024-03-25T18:27:34.022 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.018+0000 7fed277fe640 1 -- 172.21.15.97:0/3664913998 learned_addr learned my addr 172.21.15.97:0/3664913998 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:27:34.075 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.073+0000 7fed25ffb640 1 -- 172.21.15.97:0/3664913998 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1214557142 0 0) 0x7fed280af940 con 0x7fed280b4d60 2024-03-25T18:27:34.075 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.073+0000 7fed25ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:27:34.075 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.073+0000 7fed25ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:34.075 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.073+0000 7fed25ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:34.075 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.073+0000 7fed25ffb640 10 monclient(hunting): my global_id is 4728 2024-03-25T18:27:34.075 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.073+0000 7fed25ffb640 1 -- 172.21.15.97:0/3664913998 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed10003520 con 0x7fed280b4d60 2024-03-25T18:27:34.091 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.089+0000 7fed25ffb640 1 -- 172.21.15.97:0/3664913998 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4167424952 0 0) 0x7fed10003520 con 0x7fed280b4d60 2024-03-25T18:27:34.092 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.089+0000 7fed25ffb640 1 -- 172.21.15.97:0/3664913998 >> v1:172.21.15.97:6790/0 conn(0x7fed2814be90 legacy=0x7fed2814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:34.092 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.089+0000 7fed25ffb640 1 -- 172.21.15.97:0/3664913998 >> v1:172.21.15.98:6789/0 conn(0x7fed280b1730 legacy=0x7fed280b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:34.092 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.089+0000 7fed25ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:27:34.092 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.089+0000 7fed25ffb640 1 monclient: found mon.noname-a 2024-03-25T18:27:34.092 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.089+0000 7fed25ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:34.092 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.089+0000 7fed25ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.97:6789/0 2024-03-25T18:27:34.092 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.089+0000 7fed25ffb640 1 -- 172.21.15.97:0/3664913998 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed280b1030 con 0x7fed280b4d60 2024-03-25T18:27:34.092 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.089+0000 7fed25ffb640 10 monclient: _finish_auth 0 2024-03-25T18:27:34.092 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.089+0000 7fed25ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:27:34.092 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.089+0000 7fed25ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:34.092 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.089+0000 7fed25ffb640 1 -- 172.21.15.97:0/3664913998 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7fed20001a10 con 0x7fed280b4d60 2024-03-25T18:27:34.092 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.089+0000 7fed2d87d640 5 monclient: authenticate success, global_id 4728 2024-03-25T18:27:34.092 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.089+0000 7fed2d87d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:27:34.093 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.089+0000 7fed25ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:34.093 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.089+0000 7fed25ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:27:34.093 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.089+0000 7fed25ffb640 10 monclient: dump: 2024-03-25T18:27:34.093 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:27:34.093 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:27:34.093 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:27:34.093 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:27:34.093 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:34.093 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:27:34.093 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:27:34.093 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:27:34.093 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:27:34.093 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:27:34.093 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.089+0000 7fed2d87d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:27:34.117 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.116+0000 7fed25ffb640 1 -- 172.21.15.97:0/3664913998 <== mon.0 v1:172.21.15.97:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fed20001b70 con 0x7fed280b4d60 2024-03-25T18:27:34.118 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.116+0000 7fed25ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:34.118 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.116+0000 7fed25ffb640 1 -- 172.21.15.97:0/3664913998 <== mon.0 v1:172.21.15.97:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7fed20001f80 con 0x7fed280b4d60 2024-03-25T18:27:34.118 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.116+0000 7fed25ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:34.118 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.116+0000 7fed25ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:27:34.118 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.116+0000 7fed25ffb640 10 monclient: dump: 2024-03-25T18:27:34.118 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:27:34.118 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:27:34.118 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:27:34.118 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:27:34.118 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:34.118 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:27:34.119 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:27:34.119 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:27:34.119 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:27:34.119 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:27:34.119 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.116+0000 7fed2d87d640 10 monclient: get_monmap_and_config success 2024-03-25T18:27:34.119 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.116+0000 7fed2d87d640 4 set_mon_vals no callback set 2024-03-25T18:27:34.122 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.116+0000 7fed2d87d640 10 monclient: shutdown 2024-03-25T18:27:34.122 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.116+0000 7fed2d87d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:27:34.123 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.116+0000 7fed2d87d640 1 -- 172.21.15.97:0/3664913998 >> v1:172.21.15.97:6789/0 conn(0x7fed280b4d60 legacy=0x7fed2813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:34.123 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.116+0000 7fed2d87d640 1 -- 172.21.15.97:0/3664913998 shutdown_connections 2024-03-25T18:27:34.123 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.116+0000 7fed2d87d640 1 -- 172.21.15.97:0/3664913998 >> 172.21.15.97:0/3664913998 conn(0x7fed280617a0 msgr2=0x7fed28061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:34.123 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.116+0000 7fed2d87d640 1 -- 172.21.15.97:0/3664913998 shutdown_connections 2024-03-25T18:27:34.123 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.116+0000 7fed2d87d640 1 -- 172.21.15.97:0/3664913998 wait complete. 2024-03-25T18:27:34.123 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.117+0000 7fed2d87d640 10 monclient: build_initial_monmap 2024-03-25T18:27:34.123 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.117+0000 7fed2d87d640 10 monclient: monmap: 2024-03-25T18:27:34.123 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:27:34.123 INFO:teuthology.misc.health.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:27:34.123 INFO:teuthology.misc.health.smithi097.stderr:last_changed 0.000000 2024-03-25T18:27:34.124 INFO:teuthology.misc.health.smithi097.stderr:created 0.000000 2024-03-25T18:27:34.124 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:34.124 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 1 2024-03-25T18:27:34.124 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:27:34.124 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T18:27:34.124 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-c 2024-03-25T18:27:34.124 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:27:34.124 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.117+0000 7fed2d87d640 1 librados: starting msgr at 2024-03-25T18:27:34.124 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.117+0000 7fed2d87d640 1 librados: starting objecter 2024-03-25T18:27:34.124 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.117+0000 7fed2d87d640 1 Processor -- start 2024-03-25T18:27:34.124 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.118+0000 7fed2d87d640 1 -- start start 2024-03-25T18:27:34.124 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.118+0000 7fed2d87d640 1 librados: setting wanted keys 2024-03-25T18:27:34.124 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.118+0000 7fed2d87d640 1 librados: calling monclient init 2024-03-25T18:27:34.124 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.118+0000 7fed2d87d640 10 monclient: init 2024-03-25T18:27:34.125 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.118+0000 7fed2d87d640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:34.125 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.118+0000 7fed2d87d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:27:34.125 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.118+0000 7fed2d87d640 10 monclient(hunting): picked mon.noname-c con 0x7fed280b1730 addr v1:172.21.15.97:6790/0 2024-03-25T18:27:34.125 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.118+0000 7fed2d87d640 10 monclient(hunting): picked mon.noname-a con 0x7fed280b4d60 addr v1:172.21.15.97:6789/0 2024-03-25T18:27:34.125 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.118+0000 7fed2d87d640 10 monclient(hunting): picked mon.noname-b con 0x7fed2814be90 addr v1:172.21.15.98:6789/0 2024-03-25T18:27:34.125 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.118+0000 7fed2d87d640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed2813a940 con 0x7fed280b4d60 2024-03-25T18:27:34.125 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.118+0000 7fed2d87d640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed2813ab90 con 0x7fed2814be90 2024-03-25T18:27:34.125 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.118+0000 7fed2d87d640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed281d7840 con 0x7fed280b1730 2024-03-25T18:27:34.125 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.118+0000 7fed2d87d640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:34.125 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.118+0000 7fed2d87d640 10 monclient(hunting): authenticate will time out at 8266.394658s 2024-03-25T18:27:34.134 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.132+0000 7fed26ffd640 1 --1- >> v1:172.21.15.97:6790/0 conn(0x7fed280b1730 0x7fed2813bb30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.97:6790/0 says I am v1:172.21.15.97:57824/0 (socket says 172.21.15.97:57824) 2024-03-25T18:27:34.134 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.132+0000 7fed26ffd640 1 -- 172.21.15.97:0/1545810572 learned_addr learned my addr 172.21.15.97:0/1545810572 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:27:34.134 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.133+0000 7fed0ffff640 10 client.?.objecter ms_handle_connect 0x7fed280b1730 2024-03-25T18:27:34.134 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.133+0000 7fed0ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:34.139 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.137+0000 7fed0ffff640 10 client.?.objecter ms_handle_connect 0x7fed280b4d60 2024-03-25T18:27:34.139 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.137+0000 7fed0ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:34.139 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.137+0000 7fed0ffff640 10 client.?.objecter ms_handle_connect 0x7fed2814be90 2024-03-25T18:27:34.139 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.137+0000 7fed0ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:34.147 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.146+0000 7fed0ffff640 1 -- 172.21.15.97:0/1545810572 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3056477422 0 0) 0x7fed2813ab90 con 0x7fed2814be90 2024-03-25T18:27:34.147 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.146+0000 7fed0ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:27:34.147 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.146+0000 7fed0ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:34.147 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.146+0000 7fed0ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:34.148 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.146+0000 7fed0ffff640 10 monclient(hunting): my global_id is 4741 2024-03-25T18:27:34.148 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.146+0000 7fed0ffff640 1 -- 172.21.15.97:0/1545810572 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed14004eb0 con 0x7fed2814be90 2024-03-25T18:27:34.155 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.154+0000 7fed0ffff640 1 -- 172.21.15.97:0/1545810572 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1345930425 0 0) 0x7fed281d7840 con 0x7fed280b1730 2024-03-25T18:27:34.155 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.154+0000 7fed0ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:27:34.156 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.154+0000 7fed0ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:34.156 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.154+0000 7fed0ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:34.156 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.154+0000 7fed0ffff640 10 monclient(hunting): my global_id is 4712 2024-03-25T18:27:34.156 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.154+0000 7fed0ffff640 1 -- 172.21.15.97:0/1545810572 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed2813ab90 con 0x7fed280b1730 2024-03-25T18:27:34.156 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.154+0000 7fed0ffff640 1 -- 172.21.15.97:0/1545810572 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3732165958 0 0) 0x7fed2813a940 con 0x7fed280b4d60 2024-03-25T18:27:34.156 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.154+0000 7fed0ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:27:34.156 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.154+0000 7fed0ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:34.156 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.154+0000 7fed0ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:34.156 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.154+0000 7fed0ffff640 10 monclient(hunting): my global_id is 4731 2024-03-25T18:27:34.156 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.154+0000 7fed0ffff640 1 -- 172.21.15.97:0/1545810572 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed281d7840 con 0x7fed280b4d60 2024-03-25T18:27:34.171 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.169+0000 7fed0ffff640 1 -- 172.21.15.97:0/1545810572 <== mon.2 v1:172.21.15.97:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 316632686 0 0) 0x7fed2813ab90 con 0x7fed280b1730 2024-03-25T18:27:34.171 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.169+0000 7fed0ffff640 1 -- 172.21.15.97:0/1545810572 --> v1:172.21.15.97:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fed2813a940 con 0x7fed280b1730 2024-03-25T18:27:34.171 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.169+0000 7fed0ffff640 1 -- 172.21.15.97:0/1545810572 <== mon.2 v1:172.21.15.97:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7fed20001a10 con 0x7fed280b1730 2024-03-25T18:27:34.171 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.169+0000 7fed0ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:34.171 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.169+0000 7fed0ffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:27:34.171 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.169+0000 7fed0ffff640 10 monclient(hunting): dump: 2024-03-25T18:27:34.171 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:27:34.171 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:27:34.171 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:27:34.172 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:27:34.172 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:34.172 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:27:34.172 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:27:34.172 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:27:34.172 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:27:34.172 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:27:34.172 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.169+0000 7fed0ffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:27:34.174 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.173+0000 7fed0ffff640 1 -- 172.21.15.97:0/1545810572 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1209676812 0 0) 0x7fed14004eb0 con 0x7fed2814be90 2024-03-25T18:27:34.174 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.173+0000 7fed0ffff640 1 -- 172.21.15.97:0/1545810572 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fed2813ab90 con 0x7fed2814be90 2024-03-25T18:27:34.174 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.173+0000 7fed0ffff640 1 -- 172.21.15.97:0/1545810572 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7fed18001780 con 0x7fed2814be90 2024-03-25T18:27:34.174 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.173+0000 7fed0ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:34.174 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.173+0000 7fed0ffff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:27:34.175 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.173+0000 7fed0ffff640 10 monclient(hunting): dump: 2024-03-25T18:27:34.175 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:27:34.175 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:27:34.175 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:27:34.175 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:27:34.175 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:34.175 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:27:34.175 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:27:34.175 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:27:34.175 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:27:34.176 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:27:34.179 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.177+0000 7fed0ffff640 1 -- 172.21.15.97:0/1545810572 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3556547060 0 0) 0x7fed281d7840 con 0x7fed280b4d60 2024-03-25T18:27:34.179 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.177+0000 7fed0ffff640 1 -- 172.21.15.97:0/1545810572 --> v1:172.21.15.97:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fed14004eb0 con 0x7fed280b4d60 2024-03-25T18:27:34.179 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.177+0000 7fed0ffff640 1 -- 172.21.15.97:0/1545810572 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7fed1c001860 con 0x7fed280b4d60 2024-03-25T18:27:34.179 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.177+0000 7fed0ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:34.179 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.177+0000 7fed0ffff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:27:34.179 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.177+0000 7fed0ffff640 10 monclient(hunting): dump: 2024-03-25T18:27:34.179 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:27:34.179 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:27:34.179 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:27:34.179 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:27:34.179 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:34.179 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:27:34.180 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:27:34.180 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:27:34.180 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:27:34.180 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:27:34.196 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.194+0000 7fed0ffff640 1 -- 172.21.15.97:0/1545810572 <== mon.0 v1:172.21.15.97:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 638277842 0 0) 0x7fed14004eb0 con 0x7fed280b4d60 2024-03-25T18:27:34.196 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.194+0000 7fed0ffff640 1 -- 172.21.15.97:0/1545810572 >> v1:172.21.15.97:6790/0 conn(0x7fed280b1730 legacy=0x7fed2813bb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:34.197 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.194+0000 7fed0ffff640 1 -- 172.21.15.97:0/1545810572 >> v1:172.21.15.98:6789/0 conn(0x7fed2814be90 legacy=0x7fed281e3030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:34.197 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.194+0000 7fed0ffff640 10 monclient: _finish_hunting 0 2024-03-25T18:27:34.197 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.194+0000 7fed0ffff640 1 monclient: found mon.a 2024-03-25T18:27:34.197 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.194+0000 7fed0ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:34.197 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.194+0000 7fed0ffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:27:34.197 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.194+0000 7fed0ffff640 1 -- 172.21.15.97:0/1545810572 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed2813c240 con 0x7fed280b4d60 2024-03-25T18:27:34.197 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.194+0000 7fed0ffff640 10 monclient: _finish_auth 0 2024-03-25T18:27:34.197 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.194+0000 7fed0ffff640 10 monclient: _check_auth_tickets 2024-03-25T18:27:34.197 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.194+0000 7fed0ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:34.197 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.194+0000 7fed2d87d640 5 monclient: authenticate success, global_id 4731 2024-03-25T18:27:34.197 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.194+0000 7fed2d87d640 10 monclient: _renew_subs 2024-03-25T18:27:34.197 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.194+0000 7fed2d87d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:27:34.197 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.194+0000 7fed2d87d640 1 -- 172.21.15.97:0/1545810572 --> v1:172.21.15.97:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed28139ce0 con 0x7fed280b4d60 2024-03-25T18:27:34.198 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.194+0000 7fed2d87d640 10 client.4731.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:27:34.198 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.194+0000 7fed2d87d640 10 monclient: _renew_subs 2024-03-25T18:27:34.198 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.194+0000 7fed2d87d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:27:34.198 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.194+0000 7fed2d87d640 1 -- 172.21.15.97:0/1545810572 --> v1:172.21.15.97:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fed281d8650 con 0x7fed280b4d60 2024-03-25T18:27:34.198 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.194+0000 7fed2d87d640 1 librados: init done 2024-03-25T18:27:34.198 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.195+0000 7fed2d87d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:34.199 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.195+0000 7fed2d87d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:34.199 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.195+0000 7fed2d87d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:27:34.199 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.195+0000 7fed2d87d640 1 -- 172.21.15.97:0/1545810572 --> v1:172.21.15.97:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed281401f0 con 0x7fed280b4d60 2024-03-25T18:27:34.522 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.520+0000 7fed0ffff640 1 -- 172.21.15.97:0/1545810572 <== mon.0 v1:172.21.15.97:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fed1c001d30 con 0x7fed280b4d60 2024-03-25T18:27:34.522 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.520+0000 7fed0ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:34.523 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.520+0000 7fed0ffff640 1 -- 172.21.15.97:0/1545810572 <== mon.0 v1:172.21.15.97:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7fed1c003790 con 0x7fed280b4d60 2024-03-25T18:27:34.523 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.520+0000 7fed0ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:34.523 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.520+0000 7fed0ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:27:34.523 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.520+0000 7fed0ffff640 10 monclient: dump: 2024-03-25T18:27:34.523 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:27:34.523 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:27:34.523 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:27:34.523 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:27:34.523 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:34.523 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:27:34.523 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:27:34.523 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:27:34.523 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:27:34.524 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:27:34.524 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.520+0000 7fed257fa640 4 set_mon_vals no callback set 2024-03-25T18:27:34.524 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.522+0000 7fed0ffff640 1 -- 172.21.15.97:0/1545810572 <== mon.0 v1:172.21.15.97:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2377622137 0 0) 0x7fed1c0197e0 con 0x7fed280b4d60 2024-03-25T18:27:34.524 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.522+0000 7fed26ffd640 10 monclient: get_auth_request con 0x7fed1406e110 auth_method 0 2024-03-25T18:27:34.524 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.522+0000 7fed0ffff640 10 client.4731.objecter ms_handle_connect 0x7fed1406e110 2024-03-25T18:27:34.530 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.528+0000 7fed0ffff640 1 -- 172.21.15.97:0/1545810572 <== mon.0 v1:172.21.15.97:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3078631805 0 0) 0x7fed1c053570 con 0x7fed280b4d60 2024-03-25T18:27:34.530 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.528+0000 7fed0ffff640 10 client.4731.objecter ms_dispatch 0x7fed28000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:27:34.530 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.528+0000 7fed0ffff640 3 client.4731.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:27:34.530 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.528+0000 7fed0ffff640 3 client.4731.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:27:34.530 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.528+0000 7fed0ffff640 20 client.4731.objecter dump_active .. 0 homeless 2024-03-25T18:27:34.530 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.528+0000 7fed0ffff640 1 -- 172.21.15.97:0/1545810572 <== mon.0 v1:172.21.15.97:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fed1c04eeb0 con 0x7fed280b4d60 2024-03-25T18:27:34.530 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.528+0000 7fed0ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:34.531 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.528+0000 7fed0ffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:27:34.774 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.772+0000 7fed2d87d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:34.774 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.772+0000 7fed2d87d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:34.774 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.772+0000 7fed2d87d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:27:34.774 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.772+0000 7fed2d87d640 1 -- 172.21.15.97:0/1545810572 --> v1:172.21.15.97:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fed280627b0 con 0x7fed280b4d60 2024-03-25T18:27:34.810 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.808+0000 7fed0ffff640 1 -- 172.21.15.97:0/1545810572 <== mon.0 v1:172.21.15.97:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 337131954) 0x7fed1c050fb0 con 0x7fed280b4d60 2024-03-25T18:27:34.810 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.808+0000 7fed0ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:34.810 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.808+0000 7fed0ffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:27:34.810 INFO:teuthology.misc.health.smithi097.stdout: 2024-03-25T18:27:34.810 INFO:teuthology.misc.health.smithi097.stdout:{"epoch":11,"fsid":"4c3d472c-79b8-486b-9e3d-2738d4306754","created":"2024-03-25T18:16:14.737309+0000","modified":"2024-03-25T18:26:18.750062+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:18.750062+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cb5152b9-262f-45c3-94b8-18d0e81df558","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"6daf8793-0838-473d-a7d6-9a0d2894c75d","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"136f1493-3b9c-4f43-a2e7-641482d69dae","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"397d538c-4f64-4daf-9e8b-662d09f91327","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4f1df08-663f-4bda-93f8-3878ab7c24b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c4b18d70-a3ce-4b15-9b70-eae5525bd761","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8dd4033a-de0e-4f8a-96dc-59f3c1f825b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0077722e-ffa6-4a3d-9891-929d9869b286","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:27:34.812 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.810+0000 7fed2d87d640 10 librados: watch_flush enter 2024-03-25T18:27:34.812 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.810+0000 7fed2d87d640 10 librados: watch_flush exit 2024-03-25T18:27:34.812 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.810+0000 7fed2d87d640 20 client.4731.objecter shutdown clearing up homeless session... 2024-03-25T18:27:34.812 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.810+0000 7fed2d87d640 10 client.4731.objecter successfully canceled tick 2024-03-25T18:27:34.812 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.810+0000 7fed2d87d640 1 -- 172.21.15.97:0/1545810572 >> v1:172.21.15.98:6800/800387573 conn(0x7fed1406e110 legacy=0x7fed140705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:34.812 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.810+0000 7fed2d87d640 10 monclient: shutdown 2024-03-25T18:27:34.812 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.810+0000 7fed2d87d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:27:34.812 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.810+0000 7fed2d87d640 1 -- 172.21.15.97:0/1545810572 >> v1:172.21.15.97:6789/0 conn(0x7fed280b4d60 legacy=0x7fed281d7d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:34.813 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.811+0000 7fed2d87d640 1 -- 172.21.15.97:0/1545810572 shutdown_connections 2024-03-25T18:27:34.813 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.811+0000 7fed2d87d640 1 -- 172.21.15.97:0/1545810572 >> 172.21.15.97:0/1545810572 conn(0x7fed280617a0 msgr2=0x7fed28137510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:34.813 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.811+0000 7fed2d87d640 1 -- 172.21.15.97:0/1545810572 shutdown_connections 2024-03-25T18:27:34.813 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.811+0000 7fed2d87d640 1 -- 172.21.15.97:0/1545810572 wait complete. 2024-03-25T18:27:34.813 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:34.811+0000 7fed2d87d640 1 librados: shutdown 2024-03-25T18:27:34.824 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:27:41.632 DEBUG:teuthology.orchestra.run.smithi097:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:27:41.747 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.745+0000 7f0cc24af640 10 monclient: get_monmap_and_config 2024-03-25T18:27:41.747 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.745+0000 7f0cc24af640 10 monclient: build_initial_monmap 2024-03-25T18:27:41.747 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.745+0000 7f0cc24af640 10 monclient: monmap: 2024-03-25T18:27:41.748 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:27:41.748 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:27:41.748 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:27:41.747134+0000 2024-03-25T18:27:41.748 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:27:41.747134+0000 2024-03-25T18:27:41.748 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:41.748 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:27:41.748 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:27:41.748 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T18:27:41.748 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-b 2024-03-25T18:27:41.748 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:27:41.750 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.747+0000 7f0cc24af640 1 Processor -- start 2024-03-25T18:27:41.750 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.747+0000 7f0cc24af640 1 -- start start 2024-03-25T18:27:41.750 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.747+0000 7f0cc24af640 10 monclient: init 2024-03-25T18:27:41.750 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.747+0000 7f0cc24af640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:41.750 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.747+0000 7f0cc24af640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:27:41.750 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.747+0000 7f0cc24af640 10 monclient(hunting): picked mon.noname-c con 0x7f0cbc0a98f0 addr v1:172.21.15.98:6789/0 2024-03-25T18:27:41.750 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.747+0000 7f0cc24af640 10 monclient(hunting): picked mon.noname-b con 0x7f0cbc13f710 addr v1:172.21.15.97:6790/0 2024-03-25T18:27:41.750 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.748+0000 7f0cc24af640 10 monclient(hunting): picked mon.noname-a con 0x7f0cbc1430c0 addr v1:172.21.15.97:6789/0 2024-03-25T18:27:41.750 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.748+0000 7f0cc24af640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cbc142520 con 0x7f0cbc1430c0 2024-03-25T18:27:41.750 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.748+0000 7f0cc24af640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cbc0b28a0 con 0x7f0cbc0a98f0 2024-03-25T18:27:41.750 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.748+0000 7f0cc24af640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cbc0b2a30 con 0x7f0cbc13f710 2024-03-25T18:27:41.750 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.748+0000 7f0cc24af640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:41.751 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.748+0000 7f0cc24af640 10 monclient(hunting): authenticate will time out at 8274.024074s 2024-03-25T18:27:41.762 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.761+0000 7f0cbbfff640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f0cbc0a98f0 0x7f0cbc0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.97:32986/0 (socket says 172.21.15.97:32986) 2024-03-25T18:27:41.762 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.761+0000 7f0cbbfff640 1 -- 172.21.15.97:0/738329186 learned_addr learned my addr 172.21.15.97:0/738329186 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:27:41.780 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.779+0000 7f0cbb7fe640 1 -- 172.21.15.97:0/738329186 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1562426807 0 0) 0x7f0cbc0b28a0 con 0x7f0cbc0a98f0 2024-03-25T18:27:41.780 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.779+0000 7f0cbb7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:27:41.781 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.779+0000 7f0cbb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:41.781 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.779+0000 7f0cbb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:41.781 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.779+0000 7f0cbb7fe640 10 monclient(hunting): my global_id is 4744 2024-03-25T18:27:41.781 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.779+0000 7f0cbb7fe640 1 -- 172.21.15.97:0/738329186 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ca8003520 con 0x7f0cbc0a98f0 2024-03-25T18:27:41.784 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.783+0000 7f0cbb7fe640 1 -- 172.21.15.97:0/738329186 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1825985712 0 0) 0x7f0cbc142520 con 0x7f0cbc1430c0 2024-03-25T18:27:41.784 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.783+0000 7f0cbb7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:27:41.784 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.783+0000 7f0cbb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:41.784 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.783+0000 7f0cbb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:41.784 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.783+0000 7f0cbb7fe640 10 monclient(hunting): my global_id is 4734 2024-03-25T18:27:41.784 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.783+0000 7f0cbb7fe640 1 -- 172.21.15.97:0/738329186 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0cbc0b28a0 con 0x7f0cbc1430c0 2024-03-25T18:27:41.799 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.797+0000 7f0cbb7fe640 1 -- 172.21.15.97:0/738329186 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 875606886 0 0) 0x7f0cbc0b2a30 con 0x7f0cbc13f710 2024-03-25T18:27:41.799 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.797+0000 7f0cbb7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:27:41.799 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.797+0000 7f0cbb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:41.799 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.797+0000 7f0cbb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:41.800 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.797+0000 7f0cbb7fe640 10 monclient(hunting): my global_id is 4715 2024-03-25T18:27:41.800 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.797+0000 7f0cbb7fe640 1 -- 172.21.15.97:0/738329186 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0cbc142520 con 0x7f0cbc13f710 2024-03-25T18:27:41.804 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.802+0000 7f0cbb7fe640 1 -- 172.21.15.97:0/738329186 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3724060516 0 0) 0x7f0ca8003520 con 0x7f0cbc0a98f0 2024-03-25T18:27:41.804 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.802+0000 7f0cbb7fe640 1 -- 172.21.15.97:0/738329186 >> v1:172.21.15.97:6790/0 conn(0x7f0cbc13f710 legacy=0x7f0cbc1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:41.804 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.802+0000 7f0cbb7fe640 1 -- 172.21.15.97:0/738329186 >> v1:172.21.15.97:6789/0 conn(0x7f0cbc1430c0 legacy=0x7f0cbc14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:41.804 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.802+0000 7f0cbb7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:27:41.805 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.802+0000 7f0cbb7fe640 1 monclient: found mon.noname-c 2024-03-25T18:27:41.805 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.802+0000 7f0cbb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:41.805 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.802+0000 7f0cbb7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.98:6789/0 2024-03-25T18:27:41.805 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.802+0000 7f0cbb7fe640 1 -- 172.21.15.97:0/738329186 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cbc0b44c0 con 0x7f0cbc0a98f0 2024-03-25T18:27:41.805 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.802+0000 7f0cbb7fe640 10 monclient: _finish_auth 0 2024-03-25T18:27:41.805 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.802+0000 7f0cbb7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:27:41.805 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.802+0000 7f0cbb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:41.805 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.802+0000 7f0cc24af640 5 monclient: authenticate success, global_id 4744 2024-03-25T18:27:41.805 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.802+0000 7f0cbb7fe640 1 -- 172.21.15.97:0/738329186 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f0cac001690 con 0x7f0cbc0a98f0 2024-03-25T18:27:41.805 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.802+0000 7f0cbb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:41.805 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.802+0000 7f0cbb7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:27:41.805 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.802+0000 7f0cbb7fe640 10 monclient: dump: 2024-03-25T18:27:41.806 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:27:41.806 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:27:41.806 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:27:41.806 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:27:41.806 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:41.806 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:27:41.806 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:27:41.806 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:27:41.806 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:27:41.806 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:27:41.806 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.802+0000 7f0cc24af640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:27:41.814 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.812+0000 7f0cbb7fe640 1 -- 172.21.15.97:0/738329186 <== mon.1 v1:172.21.15.98:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0cac0017f0 con 0x7f0cbc0a98f0 2024-03-25T18:27:41.814 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.812+0000 7f0cbb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:41.814 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.812+0000 7f0cbb7fe640 1 -- 172.21.15.97:0/738329186 <== mon.1 v1:172.21.15.98:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f0cac001c00 con 0x7f0cbc0a98f0 2024-03-25T18:27:41.814 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.812+0000 7f0cc24af640 10 monclient: get_monmap_and_config success 2024-03-25T18:27:41.814 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.812+0000 7f0cc24af640 4 set_mon_vals no callback set 2024-03-25T18:27:41.814 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.812+0000 7f0cbb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:41.814 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.812+0000 7f0cbb7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:27:41.814 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.812+0000 7f0cbb7fe640 10 monclient: dump: 2024-03-25T18:27:41.815 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:27:41.815 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:27:41.815 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:27:41.815 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:27:41.815 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:41.815 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:27:41.815 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:27:41.815 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:27:41.815 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:27:41.815 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:27:41.818 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.813+0000 7f0cc24af640 10 monclient: shutdown 2024-03-25T18:27:41.818 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.813+0000 7f0cc24af640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:27:41.818 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.813+0000 7f0cc24af640 1 -- 172.21.15.97:0/738329186 >> v1:172.21.15.98:6789/0 conn(0x7f0cbc0a98f0 legacy=0x7f0cbc0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:41.818 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.813+0000 7f0cc24af640 1 -- 172.21.15.97:0/738329186 shutdown_connections 2024-03-25T18:27:41.818 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.813+0000 7f0cc24af640 1 -- 172.21.15.97:0/738329186 >> 172.21.15.97:0/738329186 conn(0x7f0cbc0b7ea0 msgr2=0x7f0cbc0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:41.818 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.813+0000 7f0cc24af640 1 -- 172.21.15.97:0/738329186 shutdown_connections 2024-03-25T18:27:41.818 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.813+0000 7f0cc24af640 1 -- 172.21.15.97:0/738329186 wait complete. 2024-03-25T18:27:41.818 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.814+0000 7f0cc24af640 10 monclient: build_initial_monmap 2024-03-25T18:27:41.818 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.814+0000 7f0cc24af640 10 monclient: monmap: 2024-03-25T18:27:41.818 INFO:teuthology.misc.health.smithi097.stderr:epoch 0 2024-03-25T18:27:41.818 INFO:teuthology.misc.health.smithi097.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:27:41.819 INFO:teuthology.misc.health.smithi097.stderr:last_changed 0.000000 2024-03-25T18:27:41.819 INFO:teuthology.misc.health.smithi097.stderr:created 0.000000 2024-03-25T18:27:41.819 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:41.819 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 1 2024-03-25T18:27:41.819 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.noname-a 2024-03-25T18:27:41.819 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T18:27:41.819 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.noname-c 2024-03-25T18:27:41.819 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:27:41.819 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.814+0000 7f0cc24af640 1 librados: starting msgr at 2024-03-25T18:27:41.819 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.814+0000 7f0cc24af640 1 librados: starting objecter 2024-03-25T18:27:41.819 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.815+0000 7f0cc24af640 1 Processor -- start 2024-03-25T18:27:41.819 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.815+0000 7f0cc24af640 1 -- start start 2024-03-25T18:27:41.819 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.815+0000 7f0cc24af640 1 librados: setting wanted keys 2024-03-25T18:27:41.819 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.815+0000 7f0cc24af640 1 librados: calling monclient init 2024-03-25T18:27:41.819 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.815+0000 7f0cc24af640 10 monclient: init 2024-03-25T18:27:41.820 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.815+0000 7f0cc24af640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:41.820 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.815+0000 7f0cc24af640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:27:41.820 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.815+0000 7f0cc24af640 10 monclient(hunting): picked mon.noname-c con 0x7f0cbc0a98f0 addr v1:172.21.15.97:6790/0 2024-03-25T18:27:41.820 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.815+0000 7f0cc24af640 10 monclient(hunting): picked mon.noname-a con 0x7f0cbc13f710 addr v1:172.21.15.97:6789/0 2024-03-25T18:27:41.820 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.815+0000 7f0cc24af640 10 monclient(hunting): picked mon.noname-b con 0x7f0cbc1430c0 addr v1:172.21.15.98:6789/0 2024-03-25T18:27:41.820 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.815+0000 7f0cc24af640 1 -- --> v1:172.21.15.97:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cbc139a40 con 0x7f0cbc13f710 2024-03-25T18:27:41.820 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.815+0000 7f0cc24af640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cbc133f20 con 0x7f0cbc1430c0 2024-03-25T18:27:41.820 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.815+0000 7f0cc24af640 1 -- --> v1:172.21.15.97:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cbc0ab950 con 0x7f0cbc0a98f0 2024-03-25T18:27:41.820 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.815+0000 7f0cc24af640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:41.820 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.815+0000 7f0cc24af640 10 monclient(hunting): authenticate will time out at 8274.091596s 2024-03-25T18:27:41.826 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.825+0000 7f0cbbfff640 1 --1- >> v1:172.21.15.97:6790/0 conn(0x7f0cbc0a98f0 0x7f0cbc138c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.97:6790/0 says I am v1:172.21.15.97:56628/0 (socket says 172.21.15.97:56628) 2024-03-25T18:27:41.827 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.825+0000 7f0cbbfff640 1 -- 172.21.15.97:0/4069593410 learned_addr learned my addr 172.21.15.97:0/4069593410 (peer_addr_for_me v1:172.21.15.97:0/0) 2024-03-25T18:27:41.827 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.826+0000 7f0cb97fa640 10 client.?.objecter ms_handle_connect 0x7f0cbc0a98f0 2024-03-25T18:27:41.827 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.826+0000 7f0cb97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:41.839 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.838+0000 7f0cb97fa640 10 client.?.objecter ms_handle_connect 0x7f0cbc1430c0 2024-03-25T18:27:41.840 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.838+0000 7f0cb97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:41.840 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.838+0000 7f0cb97fa640 10 client.?.objecter ms_handle_connect 0x7f0cbc13f710 2024-03-25T18:27:41.840 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.838+0000 7f0cb97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:41.846 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.844+0000 7f0cb97fa640 1 -- 172.21.15.97:0/4069593410 <== mon.2 v1:172.21.15.97:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1656407403 0 0) 0x7f0cbc0ab950 con 0x7f0cbc0a98f0 2024-03-25T18:27:41.846 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.844+0000 7f0cb97fa640 10 monclient(hunting): _negotiate 2024-03-25T18:27:41.846 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.844+0000 7f0cb97fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:41.846 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.844+0000 7f0cb97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:41.846 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.844+0000 7f0cb97fa640 10 monclient(hunting): my global_id is 4718 2024-03-25T18:27:41.846 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.844+0000 7f0cb97fa640 1 -- 172.21.15.97:0/4069593410 --> v1:172.21.15.97:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c9c004eb0 con 0x7f0cbc0a98f0 2024-03-25T18:27:41.849 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.848+0000 7f0cb97fa640 1 -- 172.21.15.97:0/4069593410 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1119720654 0 0) 0x7f0cbc133f20 con 0x7f0cbc1430c0 2024-03-25T18:27:41.849 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.848+0000 7f0cb97fa640 10 monclient(hunting): _negotiate 2024-03-25T18:27:41.849 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.848+0000 7f0cb97fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:41.849 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.848+0000 7f0cb97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:41.850 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.848+0000 7f0cb97fa640 10 monclient(hunting): my global_id is 4747 2024-03-25T18:27:41.850 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.848+0000 7f0cb97fa640 1 -- 172.21.15.97:0/4069593410 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0cbc0ab950 con 0x7f0cbc1430c0 2024-03-25T18:27:41.854 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.852+0000 7f0cb97fa640 1 -- 172.21.15.97:0/4069593410 <== mon.0 v1:172.21.15.97:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1284177530 0 0) 0x7f0cbc139a40 con 0x7f0cbc13f710 2024-03-25T18:27:41.854 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.853+0000 7f0cb97fa640 10 monclient(hunting): _negotiate 2024-03-25T18:27:41.854 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.853+0000 7f0cb97fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:41.854 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.853+0000 7f0cb97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:41.855 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.853+0000 7f0cb97fa640 10 monclient(hunting): my global_id is 4737 2024-03-25T18:27:41.855 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.853+0000 7f0cb97fa640 1 -- 172.21.15.97:0/4069593410 --> v1:172.21.15.97:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0cbc133f20 con 0x7f0cbc13f710 2024-03-25T18:27:41.863 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.862+0000 7f0cb97fa640 1 -- 172.21.15.97:0/4069593410 <== mon.0 v1:172.21.15.97:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3219841382 0 0) 0x7f0cbc133f20 con 0x7f0cbc13f710 2024-03-25T18:27:41.863 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.862+0000 7f0cb97fa640 1 -- 172.21.15.97:0/4069593410 --> v1:172.21.15.97:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0cbc139a40 con 0x7f0cbc13f710 2024-03-25T18:27:41.864 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.862+0000 7f0cb97fa640 1 -- 172.21.15.97:0/4069593410 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4030299085 0 0) 0x7f0cbc0ab950 con 0x7f0cbc1430c0 2024-03-25T18:27:41.864 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.862+0000 7f0cb97fa640 1 -- 172.21.15.97:0/4069593410 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0cbc133f20 con 0x7f0cbc1430c0 2024-03-25T18:27:41.864 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.862+0000 7f0cb97fa640 1 -- 172.21.15.97:0/4069593410 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f0cb4001b50 con 0x7f0cbc1430c0 2024-03-25T18:27:41.864 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.862+0000 7f0cb97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:41.864 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.862+0000 7f0cb97fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:27:41.865 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.862+0000 7f0cb97fa640 10 monclient(hunting): dump: 2024-03-25T18:27:41.865 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:27:41.865 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:27:41.865 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:27:41.865 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:27:41.865 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:41.865 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:27:41.865 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:27:41.865 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:27:41.865 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:27:41.865 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:27:41.865 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.862+0000 7f0cb97fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:27:41.866 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.865+0000 7f0cb97fa640 1 -- 172.21.15.97:0/4069593410 <== mon.0 v1:172.21.15.97:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f0ca40018f0 con 0x7f0cbc13f710 2024-03-25T18:27:41.866 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.865+0000 7f0cb97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:41.866 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.865+0000 7f0cb97fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:27:41.866 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.865+0000 7f0cb97fa640 10 monclient(hunting): dump: 2024-03-25T18:27:41.866 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:27:41.867 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:27:41.867 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:27:41.867 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:27:41.867 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:41.867 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:27:41.867 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:27:41.867 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:27:41.867 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:27:41.867 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:27:41.875 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.874+0000 7f0cb97fa640 1 -- 172.21.15.97:0/4069593410 <== mon.2 v1:172.21.15.97:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2310085056 0 0) 0x7f0c9c004eb0 con 0x7f0cbc0a98f0 2024-03-25T18:27:41.875 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.874+0000 7f0cb97fa640 1 -- 172.21.15.97:0/4069593410 --> v1:172.21.15.97:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0cbc0ab950 con 0x7f0cbc0a98f0 2024-03-25T18:27:41.876 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.874+0000 7f0cb97fa640 1 -- 172.21.15.97:0/4069593410 <== mon.2 v1:172.21.15.97:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f0cac001bc0 con 0x7f0cbc0a98f0 2024-03-25T18:27:41.876 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.874+0000 7f0cb97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:41.876 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.874+0000 7f0cb97fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:27:41.876 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.874+0000 7f0cb97fa640 10 monclient(hunting): dump: 2024-03-25T18:27:41.876 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:27:41.876 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:27:41.876 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:27:41.876 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:27:41.876 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:41.876 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:27:41.876 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:27:41.876 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:27:41.876 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:27:41.876 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:27:41.885 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.883+0000 7f0cb97fa640 1 -- 172.21.15.97:0/4069593410 <== mon.0 v1:172.21.15.97:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2904764274 0 0) 0x7f0cbc139a40 con 0x7f0cbc13f710 2024-03-25T18:27:41.885 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.883+0000 7f0cb97fa640 1 -- 172.21.15.97:0/4069593410 >> v1:172.21.15.97:6790/0 conn(0x7f0cbc0a98f0 legacy=0x7f0cbc138c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:41.885 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.883+0000 7f0cb97fa640 1 -- 172.21.15.97:0/4069593410 >> v1:172.21.15.98:6789/0 conn(0x7f0cbc1430c0 legacy=0x7f0cbc133810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:41.885 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.883+0000 7f0cb97fa640 10 monclient: _finish_hunting 0 2024-03-25T18:27:41.885 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.883+0000 7f0cb97fa640 1 monclient: found mon.a 2024-03-25T18:27:41.885 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.883+0000 7f0cb97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:41.885 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.883+0000 7f0cb97fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:27:41.886 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.883+0000 7f0cb97fa640 1 -- 172.21.15.97:0/4069593410 --> v1:172.21.15.97:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cbc0abc30 con 0x7f0cbc13f710 2024-03-25T18:27:41.886 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.883+0000 7f0cb97fa640 10 monclient: _finish_auth 0 2024-03-25T18:27:41.886 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.883+0000 7f0cb97fa640 10 monclient: _check_auth_tickets 2024-03-25T18:27:41.886 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.883+0000 7f0cb97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:41.886 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.883+0000 7f0cc24af640 5 monclient: authenticate success, global_id 4737 2024-03-25T18:27:41.886 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.883+0000 7f0cc24af640 10 monclient: _renew_subs 2024-03-25T18:27:41.886 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.883+0000 7f0cc24af640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:27:41.886 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.883+0000 7f0cc24af640 1 -- 172.21.15.97:0/4069593410 --> v1:172.21.15.97:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cbc0abf10 con 0x7f0cbc13f710 2024-03-25T18:27:41.886 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.883+0000 7f0cc24af640 10 client.4737.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:27:41.886 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.883+0000 7f0cc24af640 10 monclient: _renew_subs 2024-03-25T18:27:41.886 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.883+0000 7f0cc24af640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:27:41.886 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.883+0000 7f0cc24af640 1 -- 172.21.15.97:0/4069593410 --> v1:172.21.15.97:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cbc139370 con 0x7f0cbc13f710 2024-03-25T18:27:41.886 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.883+0000 7f0cc24af640 1 librados: init done 2024-03-25T18:27:41.887 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.884+0000 7f0cc24af640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:41.887 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.884+0000 7f0cc24af640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:41.887 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.884+0000 7f0cc24af640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:27:41.887 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.884+0000 7f0cc24af640 1 -- 172.21.15.97:0/4069593410 --> v1:172.21.15.97:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cbc0627b0 con 0x7f0cbc13f710 2024-03-25T18:27:41.888 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.886+0000 7f0cb97fa640 1 -- 172.21.15.97:0/4069593410 <== mon.0 v1:172.21.15.97:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0ca4003610 con 0x7f0cbc13f710 2024-03-25T18:27:41.888 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.886+0000 7f0cb97fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:41.888 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:41.886+0000 7f0cbaffd640 4 set_mon_vals no callback set 2024-03-25T18:27:42.172 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:42.170+0000 7f0cb97fa640 1 -- 172.21.15.97:0/4069593410 <== mon.0 v1:172.21.15.97:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2075326319 0 0) 0x7f0ca4003920 con 0x7f0cbc13f710 2024-03-25T18:27:42.172 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:42.170+0000 7f0cb97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:42.172 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:42.170+0000 7f0cb97fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:27:42.172 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:42.170+0000 7f0cb97fa640 10 monclient: dump: 2024-03-25T18:27:42.172 INFO:teuthology.misc.health.smithi097.stderr:epoch 1 2024-03-25T18:27:42.172 INFO:teuthology.misc.health.smithi097.stderr:fsid 4c3d472c-79b8-486b-9e3d-2738d4306754 2024-03-25T18:27:42.172 INFO:teuthology.misc.health.smithi097.stderr:last_changed 2024-03-25T18:15:25.026680+0000 2024-03-25T18:27:42.172 INFO:teuthology.misc.health.smithi097.stderr:created 2024-03-25T18:15:25.026680+0000 2024-03-25T18:27:42.172 INFO:teuthology.misc.health.smithi097.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:42.173 INFO:teuthology.misc.health.smithi097.stderr:election_strategy: 3 2024-03-25T18:27:42.173 INFO:teuthology.misc.health.smithi097.stderr:0: v1:172.21.15.97:6789/0 mon.a 2024-03-25T18:27:42.173 INFO:teuthology.misc.health.smithi097.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T18:27:42.173 INFO:teuthology.misc.health.smithi097.stderr:2: v1:172.21.15.97:6790/0 mon.c 2024-03-25T18:27:42.173 INFO:teuthology.misc.health.smithi097.stderr: 2024-03-25T18:27:42.183 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:42.181+0000 7f0cb97fa640 1 -- 172.21.15.97:0/4069593410 <== mon.0 v1:172.21.15.97:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2377622137 0 0) 0x7f0ca40198f0 con 0x7f0cbc13f710 2024-03-25T18:27:42.183 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:42.182+0000 7f0cbbfff640 10 monclient: get_auth_request con 0x7f0c9c06e0c0 auth_method 0 2024-03-25T18:27:42.184 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:42.182+0000 7f0cb97fa640 10 client.4737.objecter ms_handle_connect 0x7f0c9c06e0c0 2024-03-25T18:27:42.190 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:42.188+0000 7f0cb97fa640 1 -- 172.21.15.97:0/4069593410 <== mon.0 v1:172.21.15.97:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3078631805 0 0) 0x7f0ca4053530 con 0x7f0cbc13f710 2024-03-25T18:27:42.190 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:42.188+0000 7f0cb97fa640 10 client.4737.objecter ms_dispatch 0x7f0cbc000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:27:42.190 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:42.188+0000 7f0cb97fa640 3 client.4737.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:27:42.190 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:42.188+0000 7f0cb97fa640 3 client.4737.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:27:42.190 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:42.188+0000 7f0cb97fa640 20 client.4737.objecter dump_active .. 0 homeless 2024-03-25T18:27:42.191 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:42.188+0000 7f0cb97fa640 1 -- 172.21.15.97:0/4069593410 <== mon.0 v1:172.21.15.97:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0ca404f040 con 0x7f0cbc13f710 2024-03-25T18:27:42.191 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:42.188+0000 7f0cb97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:42.191 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:42.188+0000 7f0cb97fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:27:42.440 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:42.438+0000 7f0cc24af640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:42.440 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:42.438+0000 7f0cc24af640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:42.440 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:42.438+0000 7f0cc24af640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.97:6789/0 2024-03-25T18:27:42.440 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:42.438+0000 7f0cc24af640 1 -- 172.21.15.97:0/4069593410 --> v1:172.21.15.97:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0cbc0b7180 con 0x7f0cbc13f710 2024-03-25T18:27:42.475 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:42.474+0000 7f0cb97fa640 1 -- 172.21.15.97:0/4069593410 <== mon.0 v1:172.21.15.97:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 337131954) 0x7f0ca4050f70 con 0x7f0cbc13f710 2024-03-25T18:27:42.475 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:42.474+0000 7f0cb97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:42.476 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:42.474+0000 7f0cb97fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:27:42.476 INFO:teuthology.misc.health.smithi097.stdout: 2024-03-25T18:27:42.476 INFO:teuthology.misc.health.smithi097.stdout:{"epoch":11,"fsid":"4c3d472c-79b8-486b-9e3d-2738d4306754","created":"2024-03-25T18:16:14.737309+0000","modified":"2024-03-25T18:26:18.750062+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:18.750062+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cb5152b9-262f-45c3-94b8-18d0e81df558","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"6daf8793-0838-473d-a7d6-9a0d2894c75d","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"136f1493-3b9c-4f43-a2e7-641482d69dae","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"397d538c-4f64-4daf-9e8b-662d09f91327","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f4f1df08-663f-4bda-93f8-3878ab7c24b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c4b18d70-a3ce-4b15-9b70-eae5525bd761","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8dd4033a-de0e-4f8a-96dc-59f3c1f825b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0077722e-ffa6-4a3d-9891-929d9869b286","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:27:42.478 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:42.475+0000 7f0cc24af640 10 librados: watch_flush enter 2024-03-25T18:27:42.478 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:42.476+0000 7f0cc24af640 10 librados: watch_flush exit 2024-03-25T18:27:42.478 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:42.476+0000 7f0cc24af640 20 client.4737.objecter shutdown clearing up homeless session... 2024-03-25T18:27:42.478 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:42.476+0000 7f0cc24af640 10 client.4737.objecter successfully canceled tick 2024-03-25T18:27:42.478 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:42.476+0000 7f0cc24af640 1 -- 172.21.15.97:0/4069593410 >> v1:172.21.15.98:6800/800387573 conn(0x7f0c9c06e0c0 legacy=0x7f0c9c070580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:42.478 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:42.476+0000 7f0cc24af640 10 monclient: shutdown 2024-03-25T18:27:42.478 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:42.476+0000 7f0cc24af640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:27:42.478 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:42.476+0000 7f0cc24af640 1 -- 172.21.15.97:0/4069593410 >> v1:172.21.15.97:6789/0 conn(0x7f0cbc13f710 legacy=0x7f0cbc133100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:42.479 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:42.476+0000 7f0cc24af640 1 -- 172.21.15.97:0/4069593410 shutdown_connections 2024-03-25T18:27:42.479 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:42.476+0000 7f0cc24af640 1 -- 172.21.15.97:0/4069593410 >> 172.21.15.97:0/4069593410 conn(0x7f0cbc0b7ea0 msgr2=0x7f0cbc074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:42.479 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:42.476+0000 7f0cc24af640 1 -- 172.21.15.97:0/4069593410 shutdown_connections 2024-03-25T18:27:42.479 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:42.476+0000 7f0cc24af640 1 -- 172.21.15.97:0/4069593410 wait complete. 2024-03-25T18:27:42.479 INFO:teuthology.misc.health.smithi097.stderr:2024-03-25T18:27:42.476+0000 7f0cc24af640 1 librados: shutdown 2024-03-25T18:27:42.490 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:27:42.490 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-25T18:27:42.491 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-25T18:27:42.492 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-25T18:27:42.492 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:27:44.159 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:11:46.197 34931== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:44.166 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:11:46.205 34931== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:44.204 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:11:46.243 34931== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:44.226 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:11:46.265 34935== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:44.227 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:11:46.270 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:44.230 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:11:46.274 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:44.233 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:11:46.272 34935== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:44.267 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:11:46.305 34935== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:44.267 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:11:46.310 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:44.393 INFO:tasks.ceph.osd.0.smithi097.stderr:==00:00:11:23.900 35245== 2024-03-25T18:27:44.393 INFO:tasks.ceph.osd.0.smithi097.stderr:==00:00:11:23.900 35245== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:27:44.431 INFO:tasks.ceph.osd.0.smithi097.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:27:48.593 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:27:48.593 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0" 2024-03-25T18:27:48.594 INFO:tasks.ceph.osd.0:Stopped 2024-03-25T18:27:48.594 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-25T18:27:48.595 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:27:50.586 INFO:tasks.ceph.osd.1.smithi097.stderr:==00:00:11:30.074 35248== 2024-03-25T18:27:50.586 INFO:tasks.ceph.osd.1.smithi097.stderr:==00:00:11:30.074 35248== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:27:50.631 INFO:tasks.ceph.osd.1.smithi097.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:27:54.696 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:27:54.697 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1" 2024-03-25T18:27:54.697 INFO:tasks.ceph.osd.1:Stopped 2024-03-25T18:27:54.697 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-25T18:27:54.698 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:27:56.549 INFO:tasks.ceph.osd.2.smithi097.stderr:==00:00:11:36.049 35247== 2024-03-25T18:27:56.549 INFO:tasks.ceph.osd.2.smithi097.stderr:==00:00:11:36.049 35247== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:27:56.590 INFO:tasks.ceph.osd.2.smithi097.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:28:00.799 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:28:00.799 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2" 2024-03-25T18:28:00.800 INFO:tasks.ceph.osd.2:Stopped 2024-03-25T18:28:00.800 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-03-25T18:28:00.800 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:28:02.610 INFO:tasks.ceph.osd.3.smithi097.stderr:==00:00:11:42.118 35246== 2024-03-25T18:28:02.611 INFO:tasks.ceph.osd.3.smithi097.stderr:==00:00:11:42.118 35246== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:28:02.649 INFO:tasks.ceph.osd.3.smithi097.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:28:04.151 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:12:06.190 34931== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:04.195 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:12:06.238 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:04.200 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:12:06.239 34931== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:04.201 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:12:06.244 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:04.223 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:12:06.266 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:04.227 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:12:06.266 34931== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:04.251 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:12:06.289 34935== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:04.258 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:12:06.297 34935== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:04.286 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:12:06.324 34935== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:06.902 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:28:06.902 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3" 2024-03-25T18:28:06.903 INFO:tasks.ceph.osd.3:Stopped 2024-03-25T18:28:06.903 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-25T18:28:06.903 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:28:09.012 INFO:tasks.ceph.osd.4.smithi098.stderr:==00:00:11:48.518 35252== 2024-03-25T18:28:09.012 INFO:tasks.ceph.osd.4.smithi098.stderr:==00:00:11:48.518 35252== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:28:09.050 INFO:tasks.ceph.osd.4.smithi098.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:28:13.005 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:28:13.005 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi098 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4" 2024-03-25T18:28:13.006 INFO:tasks.ceph.osd.4:Stopped 2024-03-25T18:28:13.006 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-03-25T18:28:13.006 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:28:14.827 INFO:tasks.ceph.osd.5.smithi098.stderr:==00:00:11:54.331 35253== 2024-03-25T18:28:14.827 INFO:tasks.ceph.osd.5.smithi098.stderr:==00:00:11:54.331 35253== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:28:14.860 INFO:tasks.ceph.osd.5.smithi098.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:28:19.108 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:28:19.108 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi098 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5" 2024-03-25T18:28:19.108 INFO:tasks.ceph.osd.5:Stopped 2024-03-25T18:28:19.109 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-03-25T18:28:19.109 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:28:20.827 INFO:tasks.ceph.osd.6.smithi098.stderr:==00:00:12:00.324 35254== 2024-03-25T18:28:20.827 INFO:tasks.ceph.osd.6.smithi098.stderr:==00:00:12:00.324 35254== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:28:20.875 INFO:tasks.ceph.osd.6.smithi098.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:28:24.239 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:12:26.278 34931== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:24.242 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:12:26.281 34931== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:24.262 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:12:26.300 34931== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:24.286 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:12:26.329 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:24.289 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:12:26.332 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:24.304 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:12:26.342 34935== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:24.304 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:12:26.347 34708== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:24.308 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:12:26.346 34935== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:24.321 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:12:26.360 34935== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:25.210 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:28:25.210 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi098 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6" 2024-03-25T18:28:25.211 INFO:tasks.ceph.osd.6:Stopped 2024-03-25T18:28:25.211 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-25T18:28:25.211 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:28:26.655 INFO:tasks.ceph.osd.7.smithi098.stderr:==00:00:12:06.133 35255== 2024-03-25T18:28:26.655 INFO:tasks.ceph.osd.7.smithi098.stderr:==00:00:12:06.133 35255== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:28:26.688 INFO:tasks.ceph.osd.7.smithi098.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:28:31.313 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:28:31.313 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi098 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7" 2024-03-25T18:28:31.314 INFO:tasks.ceph.osd.7:Stopped 2024-03-25T18:28:31.314 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-25T18:28:31.314 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-03-25T18:28:31.314 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:28:31.340 INFO:tasks.ceph.mgr.y:Stopped 2024-03-25T18:28:31.341 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-25T18:28:31.341 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:28:31.370 INFO:tasks.ceph.mgr.x:Stopped 2024-03-25T18:28:31.371 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-25T18:28:31.371 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-25T18:28:31.371 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:28:31.396 INFO:tasks.ceph.mon.a.smithi097.stderr:2024-03-25T18:28:31.383+0000 9f86640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a (PID: 34900) UID: 0 2024-03-25T18:28:31.397 INFO:tasks.ceph.mon.a.smithi097.stderr:2024-03-25T18:28:31.385+0000 9f86640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-25T18:28:31.969 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:12:34.007 34931== 2024-03-25T18:28:31.969 INFO:tasks.ceph.mon.a.smithi097.stderr:==00:00:12:34.007 34931== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:28:31.978 INFO:tasks.ceph.mon.a.smithi097.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:28:37.472 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:28:37.473 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a" 2024-03-25T18:28:37.473 INFO:tasks.ceph.mon.a:Stopped 2024-03-25T18:28:37.474 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-03-25T18:28:37.474 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:28:37.489 INFO:tasks.ceph.mon.c.smithi097.stderr:2024-03-25T18:28:37.477+0000 9f86640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c (PID: 34902) UID: 0 2024-03-25T18:28:37.490 INFO:tasks.ceph.mon.c.smithi097.stderr:2024-03-25T18:28:37.478+0000 9f86640 -1 mon.c@2(peon) e1 *** Got Signal Terminated *** 2024-03-25T18:28:37.967 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:12:40.005 34935== 2024-03-25T18:28:37.967 INFO:tasks.ceph.mon.c.smithi097.stderr:==00:00:12:40.005 34935== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:28:37.977 INFO:tasks.ceph.mon.c.smithi097.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:28:43.576 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:28:43.576 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c" 2024-03-25T18:28:43.576 INFO:tasks.ceph.mon.c:Stopped 2024-03-25T18:28:43.577 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-03-25T18:28:43.577 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:28:43.594 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-03-25T18:28:43.580+0000 9f86640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b (PID: 34704) UID: 0 2024-03-25T18:28:43.594 INFO:tasks.ceph.mon.b.smithi098.stderr:2024-03-25T18:28:43.581+0000 9f86640 -1 mon.b@1(probing) e1 *** Got Signal Terminated *** 2024-03-25T18:28:44.121 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:12:46.164 34708== 2024-03-25T18:28:44.121 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:12:46.164 34708== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:28:44.130 INFO:tasks.ceph.mon.b.smithi098.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:28:49.679 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:28:49.679 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi098 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b" 2024-03-25T18:28:49.679 INFO:tasks.ceph.mon.b:Stopped 2024-03-25T18:28:49.679 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-25T18:28:49.680 DEBUG:teuthology.orchestra.run.smithi097:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(OBJECT_MISPLACED\)' | egrep -v 'slow request' | egrep -v 'overall HEALTH_' | egrep -v '\(PG_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | head -n 1 2024-03-25T18:28:49.774 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi097.front.sepia.ceph.com 2024-03-25T18:28:49.774 DEBUG:teuthology.orchestra.run.smithi097:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-25T18:28:49.885 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi097.front.sepia.ceph.com 2024-03-25T18:28:49.886 DEBUG:teuthology.orchestra.run.smithi097:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-25T18:28:49.942 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi097.front.sepia.ceph.com 2024-03-25T18:28:49.943 DEBUG:teuthology.orchestra.run.smithi097:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-25T18:28:50.011 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi097.front.sepia.ceph.com 2024-03-25T18:28:50.012 DEBUG:teuthology.orchestra.run.smithi097:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-03-25T18:28:50.076 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi098.front.sepia.ceph.com 2024-03-25T18:28:50.076 DEBUG:teuthology.orchestra.run.smithi098:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-03-25T18:28:50.239 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi098.front.sepia.ceph.com 2024-03-25T18:28:50.239 DEBUG:teuthology.orchestra.run.smithi098:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-03-25T18:28:50.338 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi098.front.sepia.ceph.com 2024-03-25T18:28:50.338 DEBUG:teuthology.orchestra.run.smithi098:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-03-25T18:28:50.395 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi098.front.sepia.ceph.com 2024-03-25T18:28:50.395 DEBUG:teuthology.orchestra.run.smithi098:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-03-25T18:28:50.462 INFO:tasks.ceph:Archiving mon data... 2024-03-25T18:28:50.464 DEBUG:teuthology.misc:Transferring archived files from smithi097:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621258/data/mon.a.tgz 2024-03-25T18:28:50.464 DEBUG:teuthology.orchestra.run.smithi097:> mktemp 2024-03-25T18:28:50.480 INFO:teuthology.orchestra.run.smithi097.stdout:/tmp/tmp.c11ysZyW7w 2024-03-25T18:28:50.481 DEBUG:teuthology.orchestra.run.smithi097:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.c11ysZyW7w 2024-03-25T18:28:50.582 DEBUG:teuthology.orchestra.run.smithi097:> sudo chmod 0666 /tmp/tmp.c11ysZyW7w 2024-03-25T18:28:50.669 DEBUG:teuthology.orchestra.remote:smithi097:/tmp/tmp.c11ysZyW7w is 133KB 2024-03-25T18:28:50.724 DEBUG:teuthology.orchestra.run.smithi097:> rm -fr /tmp/tmp.c11ysZyW7w 2024-03-25T18:28:50.738 DEBUG:teuthology.misc:Transferring archived files from smithi097:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621258/data/mon.c.tgz 2024-03-25T18:28:50.738 DEBUG:teuthology.orchestra.run.smithi097:> mktemp 2024-03-25T18:28:50.792 INFO:teuthology.orchestra.run.smithi097.stdout:/tmp/tmp.15UstjeI4f 2024-03-25T18:28:50.792 DEBUG:teuthology.orchestra.run.smithi097:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.15UstjeI4f 2024-03-25T18:28:50.880 DEBUG:teuthology.orchestra.run.smithi097:> sudo chmod 0666 /tmp/tmp.15UstjeI4f 2024-03-25T18:28:50.964 DEBUG:teuthology.orchestra.remote:smithi097:/tmp/tmp.15UstjeI4f is 133KB 2024-03-25T18:28:51.015 DEBUG:teuthology.orchestra.run.smithi097:> rm -fr /tmp/tmp.15UstjeI4f 2024-03-25T18:28:51.030 DEBUG:teuthology.misc:Transferring archived files from smithi098:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621258/data/mon.b.tgz 2024-03-25T18:28:51.030 DEBUG:teuthology.orchestra.run.smithi098:> mktemp 2024-03-25T18:28:51.048 INFO:teuthology.orchestra.run.smithi098.stdout:/tmp/tmp.3WA4oQxWFj 2024-03-25T18:28:51.049 DEBUG:teuthology.orchestra.run.smithi098:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.3WA4oQxWFj 2024-03-25T18:28:51.151 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 0666 /tmp/tmp.3WA4oQxWFj 2024-03-25T18:28:51.240 DEBUG:teuthology.orchestra.remote:smithi098:/tmp/tmp.3WA4oQxWFj is 133KB 2024-03-25T18:28:51.295 DEBUG:teuthology.orchestra.run.smithi098:> rm -fr /tmp/tmp.3WA4oQxWFj 2024-03-25T18:28:51.310 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-25T18:28:51.311 DEBUG:teuthology.orchestra.run.smithi097:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-25T18:28:51.313 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-25T18:28:51.708 DEBUG:tasks.ceph:valgrind exception message: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-25T18:28:51.956 INFO:tasks.ceph:Archiving crash dumps... 2024-03-25T18:28:51.958 DEBUG:teuthology.misc:Transferring archived files from smithi097:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621258/remote/smithi097/crash 2024-03-25T18:28:51.958 DEBUG:teuthology.orchestra.run.smithi097:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-25T18:28:51.987 DEBUG:teuthology.misc:Transferring archived files from smithi098:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621258/remote/smithi098/crash 2024-03-25T18:28:51.987 DEBUG:teuthology.orchestra.run.smithi098:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-25T18:28:52.014 INFO:tasks.ceph:Compressing logs... 2024-03-25T18:28:52.015 DEBUG:teuthology.orchestra.run.smithi097:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-25T18:28:52.031 DEBUG:teuthology.orchestra.run.smithi098:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-25T18:28:52.107 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-03-25T18:28:52.107 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35106.log 2024-03-25T18:28:52.107 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35153.log 2024-03-25T18:28:52.107 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.35106.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34902.log 2024-03-25T18:28:52.107 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.35153.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.7.log 2024-03-25T18:28:52.107 INFO:teuthology.orchestra.run.smithi098.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.35106.log.gz 2024-03-25T18:28:52.107 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.34902.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.5.log 2024-03-25T18:28:52.107 INFO:teuthology.orchestra.run.smithi098.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.35153.log.gz 2024-03-25T18:28:52.107 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/valgrind/osd.7.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.6.log 2024-03-25T18:28:52.108 INFO:teuthology.orchestra.run.smithi098.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.34902.log.gz 2024-03-25T18:28:52.108 INFO:teuthology.orchestra.run.smithi098.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.7.log.gz 2024-03-25T18:28:52.108 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/valgrind/osd.5.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mon.b.log 2024-03-25T18:28:52.108 INFO:teuthology.orchestra.run.smithi098.stderr: 72.1% -- replaced with /var/log/ceph/valgrind/osd.5.log.gz 2024-03-25T18:28:52.108 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/valgrind/osd.6.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.4.log 2024-03-25T18:28:52.108 INFO:teuthology.orchestra.run.smithi098.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.6.log.gz 2024-03-25T18:28:52.108 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/valgrind/mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35062.log 2024-03-25T18:28:52.108 INFO:teuthology.orchestra.run.smithi098.stderr: 69.6% -- replaced with /var/log/ceph/valgrind/mon.b.log.gz 2024-03-25T18:28:52.108 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/valgrind/osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-25T18:28:52.108 INFO:teuthology.orchestra.run.smithi098.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.4.log.gz 2024-03-25T18:28:52.108 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.35062.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35022.log 2024-03-25T18:28:52.108 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-03-25T18:28:52.109 INFO:teuthology.orchestra.run.smithi098.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.35062.log.gz 2024-03-25T18:28:52.109 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.35022.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34982.log 2024-03-25T18:28:52.109 INFO:teuthology.orchestra.run.smithi098.stderr: 88.1% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-25T18:28:52.109 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-03-25T18:28:52.109 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-osd.4.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.35022.log.gz 2024-03-25T18:28:52.109 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.34982.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-03-25T18:28:52.109 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-03-25T18:28:52.109 INFO:teuthology.orchestra.run.smithi098.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.34982.log.gz 2024-03-25T18:28:52.109 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34942.log 2024-03-25T18:28:52.109 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34862.log 2024-03-25T18:28:52.109 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.34942.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-25T18:28:52.109 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.34862.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-03-25T18:28:52.109 INFO:teuthology.orchestra.run.smithi098.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.34942.log.gz 2024-03-25T18:28:52.110 INFO:teuthology.orchestra.run.smithi098.stderr: 86.3%/var/log/ceph/ceph.log: -- replaced with /var/log/ceph/ceph-client.admin.34862.log.gz 2024-03-25T18:28:52.110 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-mon.b.log: 87.3% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-25T18:28:52.113 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37235.log 2024-03-25T18:28:52.113 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40571.log 2024-03-25T18:28:52.113 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.37235.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41351.log 2024-03-25T18:28:52.113 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.40571.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39592.log 2024-03-25T18:28:52.113 INFO:teuthology.orchestra.run.smithi097.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.37235.log.gz 2024-03-25T18:28:52.113 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.41351.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-03-25T18:28:52.113 INFO:teuthology.orchestra.run.smithi097.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40571.log.gz 2024-03-25T18:28:52.113 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41117.log 2024-03-25T18:28:52.113 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.39592.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.41351.log.gz 2024-03-25T18:28:52.113 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40375.log 2024-03-25T18:28:52.113 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.41117.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41976.log 2024-03-25T18:28:52.114 INFO:teuthology.orchestra.run.smithi097.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.39592.log.gz 2024-03-25T18:28:52.114 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.40375.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41819.log 2024-03-25T18:28:52.114 INFO:teuthology.orchestra.run.smithi097.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.41117.log.gz 2024-03-25T18:28:52.114 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.41976.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40727.log 2024-03-25T18:28:52.114 INFO:teuthology.orchestra.run.smithi097.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40375.log.gz 2024-03-25T18:28:52.114 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.41819.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.41976.log.gz 2024-03-25T18:28:52.114 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36260.log 2024-03-25T18:28:52.114 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.40727.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40688.log 2024-03-25T18:28:52.114 INFO:teuthology.orchestra.run.smithi097.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.41819.log.gz 2024-03-25T18:28:52.114 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.36260.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40219.log 2024-03-25T18:28:52.114 INFO:teuthology.orchestra.run.smithi097.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40727.log.gz 2024-03-25T18:28:52.114 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.40688.log: 87.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.36260.log.gz 2024-03-25T18:28:52.115 INFO:teuthology.orchestra.run.smithi097.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.40922.log 2024-03-25T18:28:52.115 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/mon.c.log/var/log/ceph/ceph-client.admin.40219.log: 2024-03-25T18:28:52.115 INFO:teuthology.orchestra.run.smithi097.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40688.log.gz 2024-03-25T18:28:52.115 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.40922.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.2.log 2024-03-25T18:28:52.115 INFO:teuthology.orchestra.run.smithi097.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.40219.log.gz 2024-03-25T18:28:52.115 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/valgrind/mon.c.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.0.log 2024-03-25T18:28:52.115 INFO:teuthology.orchestra.run.smithi097.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40922.log.gz 2024-03-25T18:28:52.115 INFO:teuthology.orchestra.run.smithi097.stderr: 69.6% -- replaced with /var/log/ceph/valgrind/mon.c.log.gz 2024-03-25T18:28:52.115 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/valgrind/osd.2.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.1.log 2024-03-25T18:28:52.115 INFO:teuthology.orchestra.run.smithi097.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.2.log.gz 2024-03-25T18:28:52.115 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/valgrind/osd.0.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mon.a.log 2024-03-25T18:28:52.116 INFO:teuthology.orchestra.run.smithi097.stderr: 72.1% -- replaced with /var/log/ceph/valgrind/osd.0.log.gz 2024-03-25T18:28:52.116 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/valgrind/osd.1.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.3.log 2024-03-25T18:28:52.116 INFO:teuthology.orchestra.run.smithi097.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.1.log.gz 2024-03-25T18:28:52.116 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/valgrind/mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40766.log 2024-03-25T18:28:52.116 INFO:teuthology.orchestra.run.smithi097.stderr: 69.6% -- replaced with /var/log/ceph/valgrind/mon.a.log.gz 2024-03-25T18:28:52.116 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/valgrind/osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38928.log 2024-03-25T18:28:52.116 INFO:teuthology.orchestra.run.smithi097.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.3.log.gz 2024-03-25T18:28:52.116 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.40766.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41312.log 2024-03-25T18:28:52.116 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.38928.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40023.log 2024-03-25T18:28:52.116 INFO:teuthology.orchestra.run.smithi097.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.40766.log.gz 2024-03-25T18:28:52.116 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.41312.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40961.log 2024-03-25T18:28:52.116 INFO:teuthology.orchestra.run.smithi097.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.38928.log.gz 2024-03-25T18:28:52.116 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.40023.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.29091.log 2024-03-25T18:28:52.117 INFO:teuthology.orchestra.run.smithi097.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.41312.log.gz 2024-03-25T18:28:52.117 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.40961.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.40023.log.gz 2024-03-25T18:28:52.117 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41156.log 2024-03-25T18:28:52.117 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph.tmp-client.admin.29091.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40844.log 2024-03-25T18:28:52.117 INFO:teuthology.orchestra.run.smithi097.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.29091.log.gz 2024-03-25T18:28:52.117 INFO:teuthology.orchestra.run.smithi097.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40961.log.gz 2024-03-25T18:28:52.117 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.41156.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39318.log 2024-03-25T18:28:52.117 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.40844.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41702.log 2024-03-25T18:28:52.117 INFO:teuthology.orchestra.run.smithi097.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.41156.log.gz 2024-03-25T18:28:52.117 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.39318.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35271.log 2024-03-25T18:28:52.117 INFO:teuthology.orchestra.run.smithi097.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.40844.log.gz 2024-03-25T18:28:52.117 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.41702.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41897.log 2024-03-25T18:28:52.117 INFO:teuthology.orchestra.run.smithi097.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39318.log.gz 2024-03-25T18:28:52.118 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.35271.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.41702.log.gzgzip 2024-03-25T18:28:52.118 INFO:teuthology.orchestra.run.smithi097.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.39670.log 2024-03-25T18:28:52.118 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.41897.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35244.log 2024-03-25T18:28:52.118 INFO:teuthology.orchestra.run.smithi097.stderr: 85.3% -- replaced with /var/log/ceph/ceph-client.admin.35271.log.gz 2024-03-25T18:28:52.118 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.39670.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38889.log 2024-03-25T18:28:52.118 INFO:teuthology.orchestra.run.smithi097.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.41897.log.gz 2024-03-25T18:28:52.118 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.35244.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41273.log 2024-03-25T18:28:52.118 INFO:teuthology.orchestra.run.smithi097.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39670.log.gz 2024-03-25T18:28:52.118 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.38889.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38967.log 2024-03-25T18:28:52.118 INFO:teuthology.orchestra.run.smithi097.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.35244.log.gz 2024-03-25T18:28:52.118 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.41273.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38809.log 2024-03-25T18:28:52.119 INFO:teuthology.orchestra.run.smithi097.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.38889.log.gz 2024-03-25T18:28:52.119 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.38967.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.41273.log.gz 2024-03-25T18:28:52.119 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39279.log 2024-03-25T18:28:52.119 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.38809.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41429.log 2024-03-25T18:28:52.119 INFO:teuthology.orchestra.run.smithi097.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.38967.log.gz 2024-03-25T18:28:52.119 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.39279.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39984.log 2024-03-25T18:28:52.119 INFO:teuthology.orchestra.run.smithi097.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.38809.log.gz 2024-03-25T18:28:52.119 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.41429.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39279.log.gz 2024-03-25T18:28:52.119 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40454.log 2024-03-25T18:28:52.119 INFO:teuthology.orchestra.run.smithi097.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.41429.log.gz 2024-03-25T18:28:52.119 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.39984.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39945.log 2024-03-25T18:28:52.119 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.40454.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41780.log 2024-03-25T18:28:52.120 INFO:teuthology.orchestra.run.smithi097.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39984.log.gz 2024-03-25T18:28:52.120 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.39945.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39709.log 2024-03-25T18:28:52.120 INFO:teuthology.orchestra.run.smithi097.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40454.log.gz 2024-03-25T18:28:52.120 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.41780.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40883.log 2024-03-25T18:28:52.120 INFO:teuthology.orchestra.run.smithi097.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.39945.log.gz 2024-03-25T18:28:52.120 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.39709.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40062.log 2024-03-25T18:28:52.120 INFO:teuthology.orchestra.run.smithi097.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.41780.log.gz 2024-03-25T18:28:52.120 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.40883.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.39709.log.gzgzip 2024-03-25T18:28:52.120 INFO:teuthology.orchestra.run.smithi097.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.40493.log 2024-03-25T18:28:52.120 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.40062.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-25T18:28:52.120 INFO:teuthology.orchestra.run.smithi097.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40883.log.gz 2024-03-25T18:28:52.120 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.40493.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41663.log 2024-03-25T18:28:52.120 INFO:teuthology.orchestra.run.smithi097.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40062.log.gz 2024-03-25T18:28:52.121 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph.audit.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40493.log.gz 2024-03-25T18:28:52.121 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39045.log 2024-03-25T18:28:52.121 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.41663.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41936.log 2024-03-25T18:28:52.121 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.39045.log: 93.4% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-25T18:28:52.121 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40415.log 2024-03-25T18:28:52.121 INFO:teuthology.orchestra.run.smithi097.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.41663.log.gz 2024-03-25T18:28:52.121 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.41936.log: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.39045.log.gz 2024-03-25T18:28:52.121 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42054.log 2024-03-25T18:28:52.121 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.40415.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39123.log 2024-03-25T18:28:52.121 INFO:teuthology.orchestra.run.smithi097.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.41936.log.gz 2024-03-25T18:28:52.121 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.42054.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41507.log 2024-03-25T18:28:52.121 INFO:teuthology.orchestra.run.smithi097.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.40415.log.gz 2024-03-25T18:28:52.122 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.39123.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.42054.log.gz 2024-03-25T18:28:52.122 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-03-25T18:28:52.122 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.41507.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39828.log 2024-03-25T18:28:52.122 INFO:teuthology.orchestra.run.smithi097.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.39123.log.gz 2024-03-25T18:28:52.122 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41468.log 2024-03-25T18:28:52.122 INFO:teuthology.orchestra.run.smithi097.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.41507.log.gz 2024-03-25T18:28:52.122 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.39828.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41624.log 2024-03-25T18:28:52.122 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.41468.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39828.log.gz 2024-03-25T18:28:52.122 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39789.log 2024-03-25T18:28:52.122 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39357.log 2024-03-25T18:28:52.122 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.41624.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41468.log.gz 2024-03-25T18:28:52.122 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.39789.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40102.log 2024-03-25T18:28:52.123 INFO:teuthology.orchestra.run.smithi097.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.41624.log.gz 2024-03-25T18:28:52.123 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.39357.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42015.log 2024-03-25T18:28:52.123 INFO:teuthology.orchestra.run.smithi097.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.39789.log.gz 2024-03-25T18:28:52.123 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.40102.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39357.log.gz 2024-03-25T18:28:52.123 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39906.log 2024-03-25T18:28:52.123 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.42015.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40297.log 2024-03-25T18:28:52.123 INFO:teuthology.orchestra.run.smithi097.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.40102.log.gz 2024-03-25T18:28:52.123 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.39906.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-03-25T18:28:52.123 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.40297.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.42015.log.gz 2024-03-25T18:28:52.123 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39475.log 2024-03-25T18:28:52.123 INFO:teuthology.orchestra.run.smithi097.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.39906.log.gz 2024-03-25T18:28:52.123 INFO:teuthology.orchestra.run.smithi097.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40297.log.gz 2024-03-25T18:28:52.124 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39397.log 2024-03-25T18:28:52.124 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.39475.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41195.log 2024-03-25T18:28:52.124 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.39397.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39553.log 2024-03-25T18:28:52.124 INFO:teuthology.orchestra.run.smithi097.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.39475.log.gz 2024-03-25T18:28:52.124 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.41195.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38850.log 2024-03-25T18:28:52.124 INFO:teuthology.orchestra.run.smithi097.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.39397.log.gz 2024-03-25T18:28:52.124 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.39553.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39084.log 2024-03-25T18:28:52.124 INFO:teuthology.orchestra.run.smithi097.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.41195.log.gz 2024-03-25T18:28:52.124 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.38850.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40180.log 2024-03-25T18:28:52.124 INFO:teuthology.orchestra.run.smithi097.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.39553.log.gz 2024-03-25T18:28:52.124 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.39084.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.38850.log.gz 2024-03-25T18:28:52.124 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39750.log 2024-03-25T18:28:52.124 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41000.log 2024-03-25T18:28:52.125 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.40180.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.39084.log.gz 2024-03-25T18:28:52.125 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.39750.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38517.log 2024-03-25T18:28:52.125 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.41000.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40180.log.gz 2024-03-25T18:28:52.125 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40649.log 2024-03-25T18:28:52.125 INFO:teuthology.orchestra.run.smithi097.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.39750.log.gz 2024-03-25T18:28:52.125 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.38517.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39631.log 2024-03-25T18:28:52.125 INFO:teuthology.orchestra.run.smithi097.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.41000.log.gz 2024-03-25T18:28:52.125 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.40649.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.38517.log.gz 2024-03-25T18:28:52.125 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39436.log 2024-03-25T18:28:52.125 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.39631.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-03-25T18:28:52.125 INFO:teuthology.orchestra.run.smithi097.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.40649.log.gz 2024-03-25T18:28:52.125 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.39436.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39006.log 2024-03-25T18:28:52.126 INFO:teuthology.orchestra.run.smithi097.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.39631.log.gz 2024-03-25T18:28:52.126 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41234.log 87.1% 2024-03-25T18:28:52.126 INFO:teuthology.orchestra.run.smithi097.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39436.log.gz 2024-03-25T18:28:52.126 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.39006.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40610.log 2024-03-25T18:28:52.126 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.41234.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.39006.log.gz 2024-03-25T18:28:52.126 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40141.log 2024-03-25T18:28:52.126 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.40610.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38770.log 2024-03-25T18:28:52.126 INFO:teuthology.orchestra.run.smithi097.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.41234.log.gz 2024-03-25T18:28:52.126 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.40141.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40610.log.gz 2024-03-25T18:28:52.126 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40805.log 2024-03-25T18:28:52.126 INFO:teuthology.orchestra.run.smithi097.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40141.log.gz 2024-03-25T18:28:52.126 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34942.log 2024-03-25T18:28:52.127 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.38770.log: gzip/var/log/ceph/ceph-client.admin.40805.log: -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-03-25T18:28:52.127 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.34942.log: 86.8%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41585.log 2024-03-25T18:28:52.127 INFO:teuthology.orchestra.run.smithi097.stderr: -- replaced with /var/log/ceph/ceph-client.admin.38770.log.gz 2024-03-25T18:28:52.127 INFO:teuthology.orchestra.run.smithi097.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.40805.log.gz 2024-03-25T18:28:52.127 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40258.log 2024-03-25T18:28:52.127 INFO:teuthology.orchestra.run.smithi097.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.34942.log.gz 2024-03-25T18:28:52.127 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.41585.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35409.log 2024-03-25T18:28:52.127 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.40258.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-25T18:28:52.127 INFO:teuthology.orchestra.run.smithi097.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.41585.log.gz 2024-03-25T18:28:52.127 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.35409.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40258.log.gz 2024-03-25T18:28:52.127 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39162.log 2024-03-25T18:28:52.127 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-03-25T18:28:52.128 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.39162.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.35409.log.gz 2024-03-25T18:28:52.128 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39867.log 2024-03-25T18:28:52.128 INFO:teuthology.orchestra.run.smithi097.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39162.log.gz 2024-03-25T18:28:52.128 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-osd.2.log: 92.9% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-25T18:28:52.128 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41078.log 2024-03-25T18:28:52.128 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.39867.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41546.log 2024-03-25T18:28:52.128 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.41078.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39867.log.gz 2024-03-25T18:28:52.128 INFO:teuthology.orchestra.run.smithi097.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.41078.log.gz 2024-03-25T18:28:52.128 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39240.log 2024-03-25T18:28:52.128 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.41546.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40336.log 2024-03-25T18:28:52.128 INFO:teuthology.orchestra.run.smithi097.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.41546.log.gz 2024-03-25T18:28:52.128 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.39240.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39240.log.gz 2024-03-25T18:28:52.129 INFO:teuthology.orchestra.run.smithi097.stderr: 93.8% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-03-25T18:28:52.129 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41390.log 2024-03-25T18:28:52.129 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5/var/log/ceph/ceph-client.admin.40336.log: --verbose -- /var/log/ceph/ceph-client.admin.39514.log 2024-03-25T18:28:52.129 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.41390.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39201.log 2024-03-25T18:28:52.129 INFO:teuthology.orchestra.run.smithi097.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.40336.log.gz 2024-03-25T18:28:52.129 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.39514.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.41390.log.gz 2024-03-25T18:28:52.129 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40532.log 2024-03-25T18:28:52.129 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.39201.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.39514.log.gz 2024-03-25T18:28:52.129 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41858.log 2024-03-25T18:28:52.129 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.40532.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41039.log 2024-03-25T18:28:52.129 INFO:teuthology.orchestra.run.smithi097.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39201.log.gz 2024-03-25T18:28:52.129 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.41858.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40532.log.gz 2024-03-25T18:28:52.129 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-03-25T18:28:52.130 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-client.admin.41039.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.41858.log.gz 2024-03-25T18:28:52.130 INFO:teuthology.orchestra.run.smithi097.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.41039.log.gz 2024-03-25T18:28:52.130 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41741.log 2024-03-25T18:28:52.130 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/ceph-mon.c.log: /var/log/ceph/ceph-client.admin.41741.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.41741.log.gz 2024-03-25T18:28:52.172 INFO:teuthology.orchestra.run.smithi098.stderr: 92.3% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-03-25T18:28:52.962 INFO:teuthology.orchestra.run.smithi098.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-03-25T18:28:52.987 INFO:teuthology.orchestra.run.smithi098.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-03-25T18:28:53.037 INFO:teuthology.orchestra.run.smithi098.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-03-25T18:28:53.268 INFO:teuthology.orchestra.run.smithi098.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-03-25T18:28:53.295 INFO:teuthology.orchestra.run.smithi097.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-03-25T18:28:53.321 INFO:teuthology.orchestra.run.smithi097.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-03-25T18:28:53.334 INFO:teuthology.orchestra.run.smithi097.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-03-25T18:28:53.336 INFO:teuthology.orchestra.run.smithi097.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-03-25T18:28:56.912 INFO:teuthology.orchestra.run.smithi097.stderr: 90.7% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-03-25T18:28:57.195 INFO:teuthology.orchestra.run.smithi097.stderr: 90.5% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-03-25T18:28:57.197 INFO:teuthology.orchestra.run.smithi097.stderr: 2024-03-25T18:28:57.198 INFO:teuthology.orchestra.run.smithi097.stderr:real 0m5.154s 2024-03-25T18:28:57.198 INFO:teuthology.orchestra.run.smithi097.stderr:user 0m14.497s 2024-03-25T18:28:57.198 INFO:teuthology.orchestra.run.smithi097.stderr:sys 0m0.544s 2024-03-25T18:28:57.376 INFO:teuthology.orchestra.run.smithi098.stderr: 90.6% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-03-25T18:28:57.378 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-25T18:28:57.378 INFO:teuthology.orchestra.run.smithi098.stderr:real 0m5.303s 2024-03-25T18:28:57.378 INFO:teuthology.orchestra.run.smithi098.stderr:user 0m8.958s 2024-03-25T18:28:57.378 INFO:teuthology.orchestra.run.smithi098.stderr:sys 0m0.324s 2024-03-25T18:28:57.378 INFO:tasks.ceph:Archiving logs... 2024-03-25T18:28:57.378 DEBUG:teuthology.misc:Transferring archived files from smithi097:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621258/remote/smithi097/log 2024-03-25T18:28:57.380 DEBUG:teuthology.orchestra.run.smithi097:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-25T18:28:59.920 DEBUG:teuthology.misc:Transferring archived files from smithi098:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621258/remote/smithi098/log 2024-03-25T18:28:59.921 DEBUG:teuthology.orchestra.run.smithi098:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-25T18:29:01.513 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 329, in valgrind_post yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-25T18:29:01.720 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=ee52c9696a2e429f9ec244200351dc23 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 329, in valgrind_post yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-25T18:29:01.724 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-25T18:29:01.733 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-25T18:29:01.771 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-25T18:29:01.771 DEBUG:teuthology.orchestra.run.smithi097:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-25T18:29:01.785 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-25T18:29:01.867 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-25T18:29:01.875 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-25T18:29:01.875 DEBUG:teuthology.orchestra.run.smithi097:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T18:29:01.878 DEBUG:teuthology.orchestra.run.smithi098:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T18:29:01.892 INFO:teuthology.orchestra.run.smithi097.stderr:bash: line 1: ntpq: command not found 2024-03-25T18:29:01.894 INFO:teuthology.orchestra.run.smithi098.stderr:bash: line 1: ntpq: command not found 2024-03-25T18:29:01.897 INFO:teuthology.orchestra.run.smithi097.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T18:29:01.898 INFO:teuthology.orchestra.run.smithi097.stdout:=============================================================================== 2024-03-25T18:29:01.898 INFO:teuthology.orchestra.run.smithi097.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 56 +770us[ +770us] +/- 67ms 2024-03-25T18:29:01.898 INFO:teuthology.orchestra.run.smithi097.stdout:^* hv02.front.sepia.ceph.com 3 6 377 61 -1130us[-1160us] +/- 52ms 2024-03-25T18:29:01.898 INFO:teuthology.orchestra.run.smithi097.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 7 +857us[ +857us] +/- 73ms 2024-03-25T18:29:01.898 INFO:teuthology.orchestra.run.smithi097.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T18:29:01.899 INFO:teuthology.orchestra.run.smithi098.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T18:29:01.899 INFO:teuthology.orchestra.run.smithi098.stdout:=============================================================================== 2024-03-25T18:29:01.900 INFO:teuthology.orchestra.run.smithi098.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 58 +782us[ +782us] +/- 67ms 2024-03-25T18:29:01.900 INFO:teuthology.orchestra.run.smithi098.stdout:^* hv02.front.sepia.ceph.com 3 6 377 58 -1107us[-1129us] +/- 52ms 2024-03-25T18:29:01.900 INFO:teuthology.orchestra.run.smithi098.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 55 +893us[ +893us] +/- 73ms 2024-03-25T18:29:01.900 INFO:teuthology.orchestra.run.smithi098.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T18:29:01.901 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-25T18:29:01.911 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-25T18:29:01.913 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-25T18:29:01.947 DEBUG:teuthology.orchestra.run.smithi097:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-25T18:29:02.019 DEBUG:teuthology.orchestra.run.smithi098:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-25T18:29:02.091 DEBUG:teuthology.orchestra.run.smithi097:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T18:29:02.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T18:29:02.123 DEBUG:teuthology.orchestra.run.smithi098:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T18:29:02.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T18:29:02.156 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-25T18:29:02.168 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-25T18:29:02.206 INFO:teuthology.task.internal:Duration was 8013.525126 seconds 2024-03-25T18:29:02.206 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-25T18:29:02.239 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-25T18:29:02.239 DEBUG:teuthology.orchestra.run.smithi097:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-25T18:29:02.242 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-25T18:29:02.285 INFO:teuthology.orchestra.run.smithi097.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T18:29:02.287 INFO:teuthology.orchestra.run.smithi098.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T18:29:02.550 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-25T18:29:02.550 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi097.front.sepia.ceph.com 2024-03-25T18:29:02.551 DEBUG:teuthology.orchestra.run.smithi097:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-25T18:29:02.573 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi098.front.sepia.ceph.com 2024-03-25T18:29:02.574 DEBUG:teuthology.orchestra.run.smithi098:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-25T18:29:02.607 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-25T18:29:02.607 DEBUG:teuthology.orchestra.run.smithi097:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-25T18:29:02.612 DEBUG:teuthology.orchestra.run.smithi098:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-25T18:29:02.678 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-25T18:29:02.679 DEBUG:teuthology.orchestra.run.smithi097:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-25T18:29:02.681 DEBUG:teuthology.orchestra.run.smithi098:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-25T18:29:02.823 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-25T18:29:02.833 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-25T18:29:02.833 DEBUG:teuthology.orchestra.run.smithi097:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-25T18:29:02.860 DEBUG:teuthology.orchestra.run.smithi098:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-25T18:29:02.887 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-25T18:29:02.897 DEBUG:teuthology.orchestra.run.smithi097:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-25T18:29:02.902 DEBUG:teuthology.orchestra.run.smithi098:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-25T18:29:02.926 INFO:teuthology.orchestra.run.smithi097.stdout:kernel.core_pattern = core 2024-03-25T18:29:02.953 INFO:teuthology.orchestra.run.smithi098.stdout:kernel.core_pattern = core 2024-03-25T18:29:02.983 DEBUG:teuthology.orchestra.run.smithi097:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-25T18:29:03.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T18:29:03.027 DEBUG:teuthology.orchestra.run.smithi098:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-25T18:29:03.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T18:29:03.041 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-25T18:29:03.051 INFO:teuthology.task.internal:Transferring archived files... 2024-03-25T18:29:03.052 DEBUG:teuthology.misc:Transferring archived files from smithi097:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621258/remote/smithi097 2024-03-25T18:29:03.052 DEBUG:teuthology.orchestra.run.smithi097:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-25T18:29:03.104 DEBUG:teuthology.misc:Transferring archived files from smithi098:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621258/remote/smithi098 2024-03-25T18:29:03.104 DEBUG:teuthology.orchestra.run.smithi098:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-25T18:29:03.141 INFO:teuthology.task.internal:Removing archive directory... 2024-03-25T18:29:03.141 DEBUG:teuthology.orchestra.run.smithi097:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-25T18:29:03.144 DEBUG:teuthology.orchestra.run.smithi098:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-25T18:29:03.188 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-25T18:29:03.199 INFO:teuthology.task.internal:Not uploading archives. 2024-03-25T18:29:03.200 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-25T18:29:03.264 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-25T18:29:03.264 DEBUG:teuthology.orchestra.run.smithi097:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-25T18:29:03.267 DEBUG:teuthology.orchestra.run.smithi098:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-25T18:29:03.279 INFO:teuthology.orchestra.run.smithi097.stdout: 265305 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 25 18:29 /home/ubuntu/cephtest 2024-03-25T18:29:03.280 INFO:teuthology.orchestra.run.smithi098.stdout: 265917 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 25 18:29 /home/ubuntu/cephtest 2024-03-25T18:29:03.282 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-25T18:29:03.297 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-25T18:29:03.364 INFO:teuthology.nuke:Checking targets against current locks 2024-03-25T18:29:03.394 DEBUG:teuthology.nuke:shortname: smithi097 2024-03-25T18:29:03.394 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T18:29:03.417 DEBUG:teuthology.nuke:shortname: smithi098 2024-03-25T18:29:03.417 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T18:29:03.421 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi097.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621258', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 16:06:47.297466', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAlUFkA3EhV5kb2ygNpF5qBDAejAgsqx9lIlhqLrnXLtG4ypJAUAbWNTJwBXqf2oFG5BOn1WPktEm7I8kPXy5/4='} 2024-03-25T18:29:03.443 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi098.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621258', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 16:06:47.295486', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBG5PMryiLKOLykE/21YnL5gEtLKYPstczuT3fgQ4kfaLo8aFaCr4KeWyCXgjdzw37FS+H3mKEFtX3vKa3yCpmuU='} 2024-03-25T18:29:03.464 INFO:teuthology.orchestra.console.smithi097:Power off 2024-03-25T18:29:03.464 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-25T18:29:03.488 DEBUG:teuthology.orchestra.console.smithi097:power off output: Chassis Power Control: Down/Off 2024-03-25T18:29:03.488 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:29:03.505 INFO:teuthology.orchestra.console.smithi098:Power off 2024-03-25T18:29:03.506 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-25T18:29:03.529 DEBUG:teuthology.orchestra.console.smithi098:power off output: Chassis Power Control: Down/Off 2024-03-25T18:29:03.530 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:29:03.540 DEBUG:teuthology.orchestra.console.smithi097:check power output: Chassis Power is on 2024-03-25T18:29:03.553 DEBUG:teuthology.orchestra.console.smithi098:check power output: Chassis Power is on 2024-03-25T18:29:07.541 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:29:07.554 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:29:07.652 DEBUG:teuthology.orchestra.console.smithi097:check power output: Chassis Power is on 2024-03-25T18:29:07.664 DEBUG:teuthology.orchestra.console.smithi098:check power output: Chassis Power is on 2024-03-25T18:29:11.657 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:29:11.666 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:29:11.767 DEBUG:teuthology.orchestra.console.smithi097:check power output: Chassis Power is on 2024-03-25T18:29:11.775 DEBUG:teuthology.orchestra.console.smithi098:check power output: Chassis Power is on 2024-03-25T18:29:15.773 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:29:15.782 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:29:15.884 DEBUG:teuthology.orchestra.console.smithi097:check power output: Chassis Power is off 2024-03-25T18:29:15.892 DEBUG:teuthology.orchestra.console.smithi098:check power output: Chassis Power is off 2024-03-25T18:29:15.986 INFO:teuthology.orchestra.console.smithi097:Power off completed 2024-03-25T18:29:15.993 INFO:teuthology.orchestra.console.smithi098:Power off completed 2024-03-25T18:29:16.098 INFO:teuthology.run:Summary data: description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} duration: 8013.525125980377 failure_reason: 'valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun' flavor: default owner: scheduled_amathuri@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=ee52c9696a2e429f9ec244200351dc23 status: fail success: false 2024-03-25T18:29:16.099 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-25T18:29:16.186 INFO:teuthology.run:FAIL